Sep 16 18:14:05.406772 ip-10-0-142-48 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:14:05.406782 ip-10-0-142-48 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:14:05.406792 ip-10-0-142-48 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:14:05.407109 ip-10-0-142-48 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:14:15.635409 ip-10-0-142-48 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:14:15.635429 ip-10-0-142-48 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 0bb30e6c844343a5ab3eb5aa019ab663 --
Sep 16 18:16:00.907889 ip-10-0-142-48 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:16:01.279396 ip-10-0-142-48 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:16:01.279396 ip-10-0-142-48 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:16:01.279396 ip-10-0-142-48 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:16:01.279396 ip-10-0-142-48 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:16:01.279396 ip-10-0-142-48 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:16:01.280363 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.280275    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:16:01.284040 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284020    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:16:01.284040 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284038    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:16:01.284040 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284043    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:16:01.284040 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284049    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284053    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284058    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284062    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284066    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284070    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284073    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284077    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284080    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284084    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284088    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284113    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284117    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284121    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284126    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284130    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284135    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284139    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284142    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284146    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:16:01.284302 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284150    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284154    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284158    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284162    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284166    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284170    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284174    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284178    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284184    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284189    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284193    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284198    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284202    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284206    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284211    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284215    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284220    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284224    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284229    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284233    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:16:01.284949 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284237    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284241    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284245    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284249    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284254    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284259    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284265    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284271    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284276    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284280    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284285    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284291    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284295    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284299    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284303    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284307    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284311    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284315    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284319    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:16:01.285499 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284323    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284327    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284332    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284337    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284342    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284345    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284350    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284355    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284359    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284364    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284368    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284372    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284377    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284384    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284389    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284394    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284399    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284403    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284409    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:16:01.286293 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284413    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284417    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284421    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284425    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284430    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284434    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284438    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284442    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284446    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284451    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284455    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284459    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284463    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284467    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284472    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284477    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284481    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.284485    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286515    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286527    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:16:01.287118 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286534    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286539    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286544    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286548    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286555    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286560    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286566    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286571    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286576    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286599    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286605    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286609    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286615    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286619    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286624    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286628    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286633    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286637    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286642    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:16:01.287996 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286647    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286651    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286655    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286660    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286664    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286668    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286672    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286676    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286681    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286686    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286690    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286698    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286703    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286708    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286713    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286718    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286722    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286726    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286730    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286735    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:16:01.288836 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286739    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286743    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286748    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286752    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286756    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286761    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286766    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286771    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286775    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286780    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286784    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286788    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286792    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286796    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286800    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286804    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286808    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286813    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286817    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:16:01.289753 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286821    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286828    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286832    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286836    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286841    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286845    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286849    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286853    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286857    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286861    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286865    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286870    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286873    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286877    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286881    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286886    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286890    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286895    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286899    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286903    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:16:01.290225 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286909    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286913    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286917    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286921    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286925    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286930    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286934    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286938    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286943    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286947    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286951    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286955    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286959    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286964    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286968    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286972    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286976    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286980    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.286985    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287602    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:16:01.290809 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287610    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287615    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287620    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287624    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287628    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287632    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287636    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287641    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287645    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287649    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287653    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287657    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287664    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287669    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287673    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287678    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287682    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287687    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287691    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:16:01.291415 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287695    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287699    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287703    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287707    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287711    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287716    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287720    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287724    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287728    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287733    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287738    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287743    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287747    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287752    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287759    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287766    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287771    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287776    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287780    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:16:01.292123 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287784    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287788    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287793    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287797    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287801    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287805    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287809    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287814    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287818    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287823    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287827    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287831    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287835    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287839    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287843    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287849    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287855    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287860    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287864    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287869    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:16:01.292604 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287873    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287878    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287883    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287887    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287891    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287898    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287903    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287907    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287910    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287914    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287918    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287922    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287927    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287931    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287935    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287939    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287944    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287948    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287952    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287956    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:16:01.293189 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287961    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287965    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287969    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287973    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287977    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287981    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287986    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287992    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.287996    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.288000    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.288005    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.288008    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.288012    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.288016    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.288020    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.288024    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.288029    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.288033    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.288037    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:16:01.293735 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.288041    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288143    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288192    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288201    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288208    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288214    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288218    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288225    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288231    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288237    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288241    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288247    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288252    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288257    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288263    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288268    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288273    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288278    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288283    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288291    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288296    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288301    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288307    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288312    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:16:01.294221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288318    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288323    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288328    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288333    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288339    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288344    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288349    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288355    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288360    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288366    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288371    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288376    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288381    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288386    2562 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288391    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288397    2562 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288402    2562 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288407    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288412    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288417    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288424    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288429    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288433    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288442    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288447    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:16:01.294812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288452    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288456    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288461    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288466    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288470    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288475    2562 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288487    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288493    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288498    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288502    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288507    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288514    2562 flags.go:64] FLAG: --help="false"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288519    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288524    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288529    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288534    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288539    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288545    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288550    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288554    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288558    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288563    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288568    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288574    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:16:01.295422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288579    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288599    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288604    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288609    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288613    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288618    2562 flags.go:64] FLAG: --lock-file=""
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288624    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288630    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288635    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288644    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288648    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288653    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288657    2562 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288662    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288667    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288672    2562 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288677    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288684    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288688    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288696    2562 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288701    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288705    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288710    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288715    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288720    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:16:01.296119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288725    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288730    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288740    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288745    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288750    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288755    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288760    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288769    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288773    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288778    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288784    2562 flags.go:64] FLAG: --port="10250"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288788    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288793    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05ea3e43d4332a4a6"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288798    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288804    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288810    2562 flags.go:64] FLAG: --register-node="true"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288815    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288821    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288825    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288830    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288834    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288840    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288844    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288849    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288853    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:16:01.296916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288857    2562 flags.go:64] FLAG: --runonce="false"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288864    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288869    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288874    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288879    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288883    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288888    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288893    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288898    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288903    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288907    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288912    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288917    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288922    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288927    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288931    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288940    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288945    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288949    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288957    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288962    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288970    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288975    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288980    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288985    2562 flags.go:64] FLAG: --v="2"
Sep 16 18:16:01.297741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288991    2562 flags.go:64] FLAG: --version="false"
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.288997    2562 flags.go:64] FLAG: --vmodule=""
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.289003    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.289008    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289191    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289198    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289203    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289208    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289214    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289219    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289223    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289227    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289232    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289236    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289240    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289244    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289248    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289252    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289257    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289261    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289265    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:16:01.298359 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289269    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289273    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289278    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289282    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289286    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289292    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289299    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289305    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289309    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289313    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289318    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289322    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289326    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289330    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289334    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289338    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289342    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289346    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289351    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:16:01.298890 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289360    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289364    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289368    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289372    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289377    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289381    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289385    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289390    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289394    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289398    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289402    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289406    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289410    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289415    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289419    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289423    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289427    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289431    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289436    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289440    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:16:01.299387 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289446    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289450    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289454    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289458    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289464    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289470    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289475    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289480    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289484    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289488    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289493    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289497    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289503    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289507    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289511    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289515    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289520    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289526    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289532    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:16:01.299939 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289537    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289542    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289547    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289553    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289559    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289563    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289568    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289573    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289576    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289581    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289601    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289605    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289610    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289615    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289619    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289623    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289627    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289632    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289637    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:16:01.300407 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289641    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:16:01.300885 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289646    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:16:01.300885 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289650    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:16:01.300885 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289654    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:16:01.300885 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.289659    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:16:01.300885 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.290379    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:16:01.300885 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.298979    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:16:01.300885 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.298995    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:16:01.300885 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.299979    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:16:01.300885 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.299986    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:16:01.300885 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300043    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:16:01.300885 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300048    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:16:01.300885 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300052    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:16:01.300885 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300055    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:16:01.300885 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300059    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:16:01.300885 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300062    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300065    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300068    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300071    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300074    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300078    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300082    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300086    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300088    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300091    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300094    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300097    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300100    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300102    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300105    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300107    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300110    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300113    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300116    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300118    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:16:01.301262 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300121    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300124    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300127    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300129    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300133    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300136    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300138    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300141    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300144    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300148    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300151    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300154    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300158    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300161    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300163    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300166    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300168    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300171    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300174    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:16:01.301813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300177    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300179    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300182    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300184    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300187    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300189    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300192    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300194    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300197    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300200    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300202    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300205    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300207    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300210    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300213    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300215    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300218    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300221    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300224    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300226    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:16:01.302270 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300229    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300232    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300234    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300237    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300240    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300242    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300245    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300247    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300250    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300252    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300255    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300257    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300260    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300262    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300264    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300268    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300270    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300273    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300275    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300278    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:16:01.302766 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300281    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:16:01.303274 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300284    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:16:01.303274 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300286    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:16:01.303274 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300290    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:16:01.303274 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300293    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:16:01.303274 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300296    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:16:01.303274 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300299    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:16:01.303274 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300302    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:16:01.303274 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300305    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:16:01.303274 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300308    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:16:01.303274 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300310    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:16:01.303274 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300313    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:16:01.303274 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300316    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:16:01.303274 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300318    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:16:01.303274 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300321    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:16:01.303274 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.300326    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300437    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300442    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300445    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300447    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300450    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300453    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300456    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300458    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300461    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300463    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300466    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300468    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300472    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300474    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300477    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300479    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300482    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300484    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300487    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300489    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:16:01.303663 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300492    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300494    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300497    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300500    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300502    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300505    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300508    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300510    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300513    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300515    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300518    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300521    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300523    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300526    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300528    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300531    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300534    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300536    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300539    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300541    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:16:01.304186 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300544    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300546    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300549    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300551    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300554    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300557    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300560    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300562    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300565    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300568    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300571    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300573    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300576    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300578    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300581    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300599    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300602    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300605    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300608    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300610    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:16:01.304679 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300613    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300616    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300618    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300621    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300623    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300626    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300630    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300634    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300636    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300639    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300641    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300644    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300646    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300649    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300651    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300654    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300656    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300659    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300662    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:16:01.305272 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300664    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300667    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300670    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300672    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300675    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300677    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300679    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300682    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300684    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300687    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300689    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300692    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300695    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300697    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300701    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300705    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300708    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300711    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:16:01.305763 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300714    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:16:01.306232 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:01.300716    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:16:01.306232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.300721    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:16:01.306232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.300824    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:16:01.306232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.303729    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:16:01.306232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.304960    2562 server.go:1019] "Starting client certificate rotation"
Sep 16 18:16:01.306232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.305073    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:16:01.306232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.305126    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:16:01.308514 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.308497    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:16:01.309608 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.309577    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:16:01.319819 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.319802    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:16:01.323615 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.323580    2562 log.go:25] "Validated CRI v1 image API"
Sep 16 18:16:01.324484 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.324468    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:16:01.327053 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.327028    2562 fs.go:135] Filesystem UUIDs: map[0d229e59-7f28-4b47-b2a0-2f0ac2ed4993:/dev/nvme0n1p4 4ccf62ff-1de4-42a6-9e23-6630c5b8bf21:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 16 18:16:01.327132 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.327051    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:16:01.333431 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.333038    2562 manager.go:216] Machine: {Timestamp:2026-09-16 18:16:01.331899613 +0000 UTC m=+0.310760689 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3046481 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28905664856df7f10a65120af11d12 SystemUUID:ec289056-6485-6df7-f10a-65120af11d12 BootID:0bb30e6c-8443-43a5-ab3e-b5aa019ab663 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:e0:b2:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:e0:b2:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:30:97:7d:de:5e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:16:01.333431 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.333412    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:16:01.333651 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.333548    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:16:01.334557 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.334528    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:16:01.337161 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.336969    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:16:01.337295 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.337164    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-48.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:16:01.337859 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.337847    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:16:01.337896 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.337862    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:16:01.337896 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.337871    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:16:01.338388 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.338378    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:16:01.339195 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.339186    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:16:01.343017 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.343005    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:16:01.343124 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.343115    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:16:01.343164 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.343144    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:16:01.343164 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.343154    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:16:01.343226 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.343167    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:16:01.343226 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.343181    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:16:01.343226 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.343189    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:16:01.343520 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.343509    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:16:01.343566 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.343531    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:16:01.345744 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.345727    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:16:01.347802 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.347789    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:16:01.347844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.347819    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:16:01.349308 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.349295    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:16:01.349376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.349311    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:16:01.349376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.349317    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:16:01.349376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.349323    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:16:01.349376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.349332    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:16:01.349376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.349337    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:16:01.349376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.349351    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:16:01.349376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.349357    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:16:01.349376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.349363    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:16:01.349376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.349369    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:16:01.349376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.349378    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:16:01.349653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.349384    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:16:01.350726 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.350716    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:16:01.350726 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.350726    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:16:01.354545 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.354531    2562 server.go:1268] "Started kubelet"
Sep 16 18:16:01.354665 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.354630    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:16:01.354717 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.354686    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:16:01.354763 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.354736    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:16:01.355118 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.355097    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:16:01.355195 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.355119    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-48.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:16:01.355195 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.355169    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:16:01.355834 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.355817    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:16:01.355993 ip-10-0-142-48 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:16:01.357260 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.357245    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:16:01.359020 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.359006    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:16:01.362254 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.361300    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-48.ec2.internal.18d5dfdba1c4d3f2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-48.ec2.internal,UID:ip-10-0-142-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-48.ec2.internal,},FirstTimestamp:2026-09-16 18:16:01.354503154 +0000 UTC m=+0.333364229,LastTimestamp:2026-09-16 18:16:01.354503154 +0000 UTC m=+0.333364229,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-48.ec2.internal,}"
Sep 16 18:16:01.363469 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.363452    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:16:01.363926 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.363912    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:16:01.363989 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.363973    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-48.ec2.internal\" not found"
Sep 16 18:16:01.364550 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.364535    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:16:01.364683 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.364672    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:16:01.364798 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.364551    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:16:01.364880 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.364868    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:16:01.364924 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.364880    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:16:01.364924 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.364903    2562 factory.go:55] Registering systemd factory
Sep 16 18:16:01.364924 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.364919    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:16:01.365119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.365107    2562 factory.go:153] Registering CRI-O factory
Sep 16 18:16:01.365159 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.365123    2562 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:16:01.365207 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.365168    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:16:01.365207 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.365186    2562 factory.go:103] Registering Raw factory
Sep 16 18:16:01.365207 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.365196    2562 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:16:01.366172 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.366159    2562 manager.go:349] Starting recovery of all containers
Sep 16 18:16:01.369610 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.369561    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-48.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:16:01.369610 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.369603    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:16:01.378304 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.378147    2562 manager.go:354] Recovery completed
Sep 16 18:16:01.382371 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.382358    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:16:01.384174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.384149    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:16:01.384251 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.384192    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:16:01.384251 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.384206    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:16:01.384754 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.384741    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:16:01.384754 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.384752    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:16:01.384893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.384759    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:16:01.384893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.384770    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:16:01.385451 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.385436    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:16:01.385526 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.385464    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:16:01.385526 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.385478    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:16:01.386437 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.386364    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-48.ec2.internal.18d5dfdba38995ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-48.ec2.internal,UID:ip-10-0-142-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-48.ec2.internal,},FirstTimestamp:2026-09-16 18:16:01.38417505 +0000 UTC m=+0.363036128,LastTimestamp:2026-09-16 18:16:01.38417505 +0000 UTC m=+0.363036128,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-48.ec2.internal,}"
Sep 16 18:16:01.387164 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.387151    2562 policy_none.go:49] "None policy: Start"
Sep 16 18:16:01.387240 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.387168    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:16:01.387240 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.387181    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:16:01.389275 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.389258    2562 policy_none.go:47] "Start"
Sep 16 18:16:01.395972 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.395914    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-48.ec2.internal.18d5dfdba389f088  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-48.ec2.internal,UID:ip-10-0-142-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-48.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-48.ec2.internal,},FirstTimestamp:2026-09-16 18:16:01.38419828 +0000 UTC m=+0.363059356,LastTimestamp:2026-09-16 18:16:01.38419828 +0000 UTC m=+0.363059356,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-48.ec2.internal,}"
Sep 16 18:16:01.405743 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.405724    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tjrr4"
Sep 16 18:16:01.407925 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.407861    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-48.ec2.internal.18d5dfdba38a2804  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-48.ec2.internal,UID:ip-10-0-142-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-48.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-48.ec2.internal,},FirstTimestamp:2026-09-16 18:16:01.384212484 +0000 UTC m=+0.363073560,LastTimestamp:2026-09-16 18:16:01.384212484 +0000 UTC m=+0.363073560,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-48.ec2.internal,}"
Sep 16 18:16:01.415093 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.415075    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tjrr4"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.416603    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-48.ec2.internal.18d5dfdba38995ca\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-48.ec2.internal.18d5dfdba38995ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-48.ec2.internal,UID:ip-10-0-142-48.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-48.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-48.ec2.internal,},FirstTimestamp:2026-09-16 18:16:01.38417505 +0000 UTC m=+0.363036128,LastTimestamp:2026-09-16 18:16:01.385449567 +0000 UTC m=+0.364310647,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-48.ec2.internal,}"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.420255    2562 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.420291    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.420303    2562 server.go:87] "Starting device plugin registration server"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.420627    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.420639    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.420727    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.420815    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.420824    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.422347    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.422400    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-48.ec2.internal\" not found"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.428071    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.429205    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.429244    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.429263    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.429334    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:16:01.443281 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.432823    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:16:01.521117 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.521096    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:16:01.522273 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.522253    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:16:01.522362 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.522289    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:16:01.522362 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.522303    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:16:01.522362 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.522333    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.529467 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.529373    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-48.ec2.internal"]
Sep 16 18:16:01.529597 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.529477    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:16:01.530985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.530968    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:16:01.531062 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.531000    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:16:01.531062 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.531016    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:16:01.533198 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.533186    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:16:01.533322 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.533307    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.533382 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.533340    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:16:01.533546 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.533531    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.533581 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.533559    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-48.ec2.internal\": node \"ip-10-0-142-48.ec2.internal\" not found"
Sep 16 18:16:01.533947 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.533932    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:16:01.534024 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.533949    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:16:01.534024 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.533957    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:16:01.534024 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.533971    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:16:01.534024 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.533973    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:16:01.534024 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.533987    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:16:01.537620 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.537576    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.537698 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.537624    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:16:01.538477 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.538459    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:16:01.538563 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.538490    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:16:01.538563 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.538505    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:16:01.566171 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.566151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5adff8773b3a030a2e2de8635bd6ee73-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal\" (UID: \"5adff8773b3a030a2e2de8635bd6ee73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.566249 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.566177    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5adff8773b3a030a2e2de8635bd6ee73-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal\" (UID: \"5adff8773b3a030a2e2de8635bd6ee73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.566249 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.566194    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f512f1af51c00a31a72cfdbb64042241-config\") pod \"kube-apiserver-proxy-ip-10-0-142-48.ec2.internal\" (UID: \"f512f1af51c00a31a72cfdbb64042241\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.573186 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.573170    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-48.ec2.internal\" not found" node="ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.574477 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.574455    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-48.ec2.internal\" not found"
Sep 16 18:16:01.577665 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.577650    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-48.ec2.internal\" not found" node="ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.666327 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.666300    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5adff8773b3a030a2e2de8635bd6ee73-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal\" (UID: \"5adff8773b3a030a2e2de8635bd6ee73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.666327 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.666327    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5adff8773b3a030a2e2de8635bd6ee73-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal\" (UID: \"5adff8773b3a030a2e2de8635bd6ee73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.666477 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.666351    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f512f1af51c00a31a72cfdbb64042241-config\") pod \"kube-apiserver-proxy-ip-10-0-142-48.ec2.internal\" (UID: \"f512f1af51c00a31a72cfdbb64042241\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.666477 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.666380    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5adff8773b3a030a2e2de8635bd6ee73-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal\" (UID: \"5adff8773b3a030a2e2de8635bd6ee73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.666477 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.666401    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5adff8773b3a030a2e2de8635bd6ee73-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal\" (UID: \"5adff8773b3a030a2e2de8635bd6ee73\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.666477 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.666421    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f512f1af51c00a31a72cfdbb64042241-config\") pod \"kube-apiserver-proxy-ip-10-0-142-48.ec2.internal\" (UID: \"f512f1af51c00a31a72cfdbb64042241\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.675418 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.675399    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-48.ec2.internal\" not found"
Sep 16 18:16:01.775960 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.775923    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-48.ec2.internal\" not found"
Sep 16 18:16:01.875778 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.875735    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.876816 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.876798    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-48.ec2.internal\" not found"
Sep 16 18:16:01.881644 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:01.881631    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-48.ec2.internal"
Sep 16 18:16:01.977696 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:01.977660    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-48.ec2.internal\" not found"
Sep 16 18:16:02.078159 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:02.078136    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-48.ec2.internal\" not found"
Sep 16 18:16:02.178744 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:02.178684    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-48.ec2.internal\" not found"
Sep 16 18:16:02.262052 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:02.262017    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:16:02.264647 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:02.264627    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-48.ec2.internal"
Sep 16 18:16:02.278102 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:02.278084    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:16:02.279522 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:02.279505    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal"
Sep 16 18:16:02.290799 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:02.290780    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:16:02.295284 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:02.295271    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:16:03.419359 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.419339    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:16:03.419470 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.419434    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:16:03.419470 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.419460    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:16:03.419559 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.419482    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:16:03.457360 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.457328    2562 apiserver.go:52] "Watching apiserver"
Sep 16 18:16:03.457577 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.457385    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:16:03.464404 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.464379    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:16:03.466082 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.466061    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-gws9r","openshift-network-operator/iptables-alerter-82fx4","openshift-ovn-kubernetes/ovnkube-node-4wwpr","kube-system/global-pull-secret-syncer-ks9dh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4","openshift-cluster-node-tuning-operator/tuned-7czp6","openshift-dns/node-resolver-x5xdc","openshift-image-registry/node-ca-65kl9","openshift-multus/multus-additional-cni-plugins-52wqd","openshift-multus/network-metrics-daemon-s9p65","openshift-network-diagnostics/network-check-target-rb6xd","kube-system/konnectivity-agent-gmfcs","kube-system/kube-apiserver-proxy-ip-10-0-142-48.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal"]
Sep 16 18:16:03.468400 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.468381    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:03.468497 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:03.468476    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:03.470413 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.470397    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:03.470483 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:03.470465    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:03.472633 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.472609    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.474920 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.474902    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:03.474995 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:03.474960    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:03.477027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.477008    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.478049 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.478029    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:16:03.478202 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.478181    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:16:03.478202 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.478195    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:16:03.478553 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.478524    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ngx8k\""
Sep 16 18:16:03.478684 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.478671    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:16:03.478748 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.478728    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:16:03.478794 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.478729    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:16:03.479219 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.479202    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.479299 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.479280    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:16:03.480318 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.480301    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:16:03.481729 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.481708    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:16:03.481822 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.481741    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zd5ks\""
Sep 16 18:16:03.482018 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.481993    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.483802 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.483782    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:16:03.483890 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.483836    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:16:03.483951 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.483836    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:16:03.483951 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.483904    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-42gkn\""
Sep 16 18:16:03.484172 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.484153    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-sysconfig\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.484232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.484181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-var-lib-kubelet\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.484232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.484200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-etc-openvswitch\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.484316 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.484250    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-65kl9"
Sep 16 18:16:03.484316 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.484242    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-cni-netd\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.484316 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.484304    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e394908-d4ac-40f4-b8ad-232904cc2d64-ovnkube-config\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.484725    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-tuned\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.484771    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdzgc\" (UniqueName: \"kubernetes.io/projected/a7725baf-5e80-4d34-9d37-43fb17f80296-kube-api-access-bdzgc\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.484802    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-run-netns\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.484831    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-run-openvswitch\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.484843    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.484865    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-run-ovn\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.484887    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-node-log\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.484917    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.484968    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e394908-d4ac-40f4-b8ad-232904cc2d64-ovn-node-metrics-cert\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485004    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89l5d\" (UniqueName: \"kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d\") pod \"network-check-target-rb6xd\" (UID: \"773c8899-dcd1-4682-9732-7761f617a2a5\") " pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485043    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-kubernetes\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485070    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-device-dir\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-etc-selinux\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485130    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-sys\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485174    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-lib-modules\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485206    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-run-systemd\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.485481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485385    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-log-socket\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485494    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-registration-dir\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485532    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-systemd\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485577    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-run-ovn-kubernetes\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485607    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-cni-bin\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485649    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmpq6\" (UniqueName: \"kubernetes.io/projected/6e394908-d4ac-40f4-b8ad-232904cc2d64-kube-api-access-bmpq6\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485679    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a30f7edc-614c-48bc-9559-c7683cc316c0-kubelet-config\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485708    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a30f7edc-614c-48bc-9559-c7683cc316c0-dbus\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485768    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-modprobe-d\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.485808    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-run\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486109    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs\") pod \"network-metrics-daemon-s9p65\" (UID: \"d93e613d-563c-4b51-93c6-4b62fb3d39aa\") " pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486183    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e394908-d4ac-40f4-b8ad-232904cc2d64-env-overrides\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486212    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-sys-fs\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486247    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-kubelet\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-slash\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.486321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486326    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-var-lib-openvswitch\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.487016 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486365    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:03.487016 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486403    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48tqw\" (UniqueName: \"kubernetes.io/projected/d93e613d-563c-4b51-93c6-4b62fb3d39aa-kube-api-access-48tqw\") pod \"network-metrics-daemon-s9p65\" (UID: \"d93e613d-563c-4b51-93c6-4b62fb3d39aa\") " pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:03.487016 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-sysctl-conf\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.487016 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486478    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-host\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.487016 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486507    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-sysctl-d\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.487016 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486529    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7725baf-5e80-4d34-9d37-43fb17f80296-tmp\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.487016 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486599    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-systemd-units\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.487016 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486634    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e394908-d4ac-40f4-b8ad-232904cc2d64-ovnkube-script-lib\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.487016 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-socket-dir\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.487016 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.486704    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wkp5\" (UniqueName: \"kubernetes.io/projected/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-kube-api-access-6wkp5\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.487760 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.487663    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:16:03.487760 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.487713    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-82fx4"
Sep 16 18:16:03.489277 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.489258    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:16:03.489630 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.489613    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:16:03.489718 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.489650    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:16:03.489789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.489771    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.492236 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.492219    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gmfcs"
Sep 16 18:16:03.493915 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.493899    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:16:03.494340 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.494323    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x5xdc"
Sep 16 18:16:03.494524 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.494510    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2m6ld\""
Sep 16 18:16:03.496931 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.496915    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:16:03.497019 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.496965    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:16:03.497174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.497160    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:16:03.497253 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.497238    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p26w7\""
Sep 16 18:16:03.499869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.499854    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:16:03.500164 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.500149    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wt6kq\""
Sep 16 18:16:03.500225 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.500203    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:16:03.500357 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.500337    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:16:03.500424 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.500406    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2fpw6\""
Sep 16 18:16:03.500479 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.500451    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:16:03.500636 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.500618    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:16:03.500636 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.500628    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vzl68\""
Sep 16 18:16:03.500793 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.500631    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vzcdn\""
Sep 16 18:16:03.500907 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.500893    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:16:03.501426 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.501413    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:16:03.508188 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.508165    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:16:03.523461 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.523435    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sk2kl"
Sep 16 18:16:03.533850 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.533792    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:11:02 +0000 UTC" deadline="2028-04-21 07:34:19.302430891 +0000 UTC"
Sep 16 18:16:03.533850 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.533848    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13981h18m15.76858537s"
Sep 16 18:16:03.533975 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.533838    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sk2kl"
Sep 16 18:16:03.561570 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:03.561516    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5adff8773b3a030a2e2de8635bd6ee73.slice/crio-d7d3ff30fe46083977f058ec59c469c3bc724cd1938fdeaf465d9a4b1c9ab68a WatchSource:0}: Error finding container d7d3ff30fe46083977f058ec59c469c3bc724cd1938fdeaf465d9a4b1c9ab68a: Status 404 returned error can't find the container with id d7d3ff30fe46083977f058ec59c469c3bc724cd1938fdeaf465d9a4b1c9ab68a
Sep 16 18:16:03.562428 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:03.562400    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf512f1af51c00a31a72cfdbb64042241.slice/crio-c2d75e9dcd6c50b240e0386991755801b13c02a4a5c04ea27a147294d1f7878f WatchSource:0}: Error finding container c2d75e9dcd6c50b240e0386991755801b13c02a4a5c04ea27a147294d1f7878f: Status 404 returned error can't find the container with id c2d75e9dcd6c50b240e0386991755801b13c02a4a5c04ea27a147294d1f7878f
Sep 16 18:16:03.566744 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.566730    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:16:03.579318 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.579300    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:16:03.587297 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587267    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e04d0991-aae5-4412-850d-6e4685c0124a-host\") pod \"node-ca-65kl9\" (UID: \"e04d0991-aae5-4412-850d-6e4685c0124a\") " pod="openshift-image-registry/node-ca-65kl9"
Sep 16 18:16:03.587395 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587306    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.587395 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587324    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-modprobe-d\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.587395 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587339    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-multus-socket-dir-parent\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.587395 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587358    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvpf9\" (UniqueName: \"kubernetes.io/projected/d5d1838b-753e-4591-b50a-473909c62656-kube-api-access-dvpf9\") pod \"node-resolver-x5xdc\" (UID: \"d5d1838b-753e-4591-b50a-473909c62656\") " pod="openshift-dns/node-resolver-x5xdc"
Sep 16 18:16:03.587395 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587386    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-slash\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.587607 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587407    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.587607 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587418    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:03.587607 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-modprobe-d\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.587607 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587442    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-slash\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.587607 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb9d634a-bee5-4019-8a89-32065530f04c-cni-binary-copy\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.587607 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:03.587554    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:03.587607 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587580    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-multus-cni-dir\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.587893 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:03.587617    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret podName:a30f7edc-614c-48bc-9559-c7683cc316c0 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:04.087590832 +0000 UTC m=+1.952494859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret") pod "global-pull-secret-syncer-ks9dh" (UID: "a30f7edc-614c-48bc-9559-c7683cc316c0") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:03.587893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587644    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-run-netns\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.587893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587664    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-sysctl-d\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.587893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587681    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7725baf-5e80-4d34-9d37-43fb17f80296-tmp\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.587893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587696    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-systemd-units\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.587893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587711    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e394908-d4ac-40f4-b8ad-232904cc2d64-ovnkube-script-lib\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.587893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587726    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-var-lib-kubelet\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.587893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587783    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-systemd-units\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.587893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587791    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-sysconfig\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.587893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e394908-d4ac-40f4-b8ad-232904cc2d64-ovnkube-config\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.587893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587840    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-sysctl-d\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.587893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587845    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-sysconfig\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.587893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587878    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-var-lib-cni-multus\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587904    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-multus-conf-dir\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4trzg\" (UniqueName: \"kubernetes.io/projected/d32eeb41-3e42-43f2-b233-339e7e76ad67-kube-api-access-4trzg\") pod \"iptables-alerter-82fx4\" (UID: \"d32eeb41-3e42-43f2-b233-339e7e76ad67\") " pod="openshift-network-operator/iptables-alerter-82fx4"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587957    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-tuned\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.587981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdzgc\" (UniqueName: \"kubernetes.io/projected/a7725baf-5e80-4d34-9d37-43fb17f80296-kube-api-access-bdzgc\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-run-openvswitch\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588028    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-run-ovn\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588054    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588059    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-run-openvswitch\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588083    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d5d1838b-753e-4591-b50a-473909c62656-hosts-file\") pod \"node-resolver-x5xdc\" (UID: \"d5d1838b-753e-4591-b50a-473909c62656\") " pod="openshift-dns/node-resolver-x5xdc"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588107    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-run-ovn\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588087    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588112    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb9d634a-bee5-4019-8a89-32065530f04c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588151    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588157    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-device-dir\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588196    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb9d634a-bee5-4019-8a89-32065530f04c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588223    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb9d634a-bee5-4019-8a89-32065530f04c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.588486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588249    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb9d634a-bee5-4019-8a89-32065530f04c-system-cni-dir\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588274    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-multus-daemon-config\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588268    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-device-dir\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d32eeb41-3e42-43f2-b233-339e7e76ad67-iptables-alerter-script\") pod \"iptables-alerter-82fx4\" (UID: \"d32eeb41-3e42-43f2-b233-339e7e76ad67\") " pod="openshift-network-operator/iptables-alerter-82fx4"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588310    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6e394908-d4ac-40f4-b8ad-232904cc2d64-ovnkube-script-lib\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588311    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6e394908-d4ac-40f4-b8ad-232904cc2d64-ovnkube-config\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588367    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-registration-dir\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588418    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-registration-dir\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-run-ovn-kubernetes\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmpq6\" (UniqueName: \"kubernetes.io/projected/6e394908-d4ac-40f4-b8ad-232904cc2d64-kube-api-access-bmpq6\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588479    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-run-ovn-kubernetes\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588498    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a30f7edc-614c-48bc-9559-c7683cc316c0-dbus\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588517    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-hostroot\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588531    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-run-multus-certs\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588561    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-run\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588567    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a30f7edc-614c-48bc-9559-c7683cc316c0-dbus\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588591    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs\") pod \"network-metrics-daemon-s9p65\" (UID: \"d93e613d-563c-4b51-93c6-4b62fb3d39aa\") " pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:03.589174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588604    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-run\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588616    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e394908-d4ac-40f4-b8ad-232904cc2d64-env-overrides\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588638    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-cnibin\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588668    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7-konnectivity-ca\") pod \"konnectivity-agent-gmfcs\" (UID: \"d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7\") " pod="kube-system/konnectivity-agent-gmfcs"
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:03.588676    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588698    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d1838b-753e-4591-b50a-473909c62656-tmp-dir\") pod \"node-resolver-x5xdc\" (UID: \"d5d1838b-753e-4591-b50a-473909c62656\") " pod="openshift-dns/node-resolver-x5xdc"
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:03.588749    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs podName:d93e613d-563c-4b51-93c6-4b62fb3d39aa nodeName:}" failed. No retries permitted until 2026-09-16 18:16:04.088735033 +0000 UTC m=+1.953639061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs") pod "network-metrics-daemon-s9p65" (UID: "d93e613d-563c-4b51-93c6-4b62fb3d39aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.588741    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9ssl\" (UniqueName: \"kubernetes.io/projected/e04d0991-aae5-4412-850d-6e4685c0124a-kube-api-access-b9ssl\") pod \"node-ca-65kl9\" (UID: \"e04d0991-aae5-4412-850d-6e4685c0124a\") " pod="openshift-image-registry/node-ca-65kl9"
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589185    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-sys-fs\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589202    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-kubelet\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589217    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-var-lib-openvswitch\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589234    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb9d634a-bee5-4019-8a89-32065530f04c-cnibin\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589249    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-var-lib-cni-bin\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48tqw\" (UniqueName: \"kubernetes.io/projected/d93e613d-563c-4b51-93c6-4b62fb3d39aa-kube-api-access-48tqw\") pod \"network-metrics-daemon-s9p65\" (UID: \"d93e613d-563c-4b51-93c6-4b62fb3d39aa\") " pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589272    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-kubelet\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-sysctl-conf\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589288    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-sys-fs\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.590015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589300    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-var-lib-openvswitch\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589307    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-host\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589361    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-run-k8s-cni-cncf-io\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589385    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-etc-kubernetes\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e04d0991-aae5-4412-850d-6e4685c0124a-serviceca\") pod \"node-ca-65kl9\" (UID: \"e04d0991-aae5-4412-850d-6e4685c0124a\") " pod="openshift-image-registry/node-ca-65kl9"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589420    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-host\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589422    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-sysctl-conf\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-socket-dir\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wkp5\" (UniqueName: \"kubernetes.io/projected/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-kube-api-access-6wkp5\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589485    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-var-lib-kubelet\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589534    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-var-lib-kubelet\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589579    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-etc-openvswitch\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589611    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6e394908-d4ac-40f4-b8ad-232904cc2d64-env-overrides\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589604    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-socket-dir\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-cni-netd\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589657    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-etc-openvswitch\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589667    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-cni-binary-copy\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.590789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589690    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-cni-netd\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589692    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-run-netns\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589723    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-node-log\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589729    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-run-netns\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e394908-d4ac-40f4-b8ad-232904cc2d64-ovn-node-metrics-cert\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589755    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-node-log\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d32eeb41-3e42-43f2-b233-339e7e76ad67-host-slash\") pod \"iptables-alerter-82fx4\" (UID: \"d32eeb41-3e42-43f2-b233-339e7e76ad67\") " pod="openshift-network-operator/iptables-alerter-82fx4"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89l5d\" (UniqueName: \"kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d\") pod \"network-check-target-rb6xd\" (UID: \"773c8899-dcd1-4682-9732-7761f617a2a5\") " pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589810    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-kubernetes\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589841    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-etc-selinux\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589862    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-sys\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589882    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-lib-modules\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589902    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-kubernetes\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589906    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-run-systemd\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-run-systemd\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589939    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-etc-selinux\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589939    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-log-socket\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.591604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-cni-bin\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.592103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589985    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-log-socket\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.592103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.589988    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-sys\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.592103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.590018    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6e394908-d4ac-40f4-b8ad-232904cc2d64-host-cni-bin\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.592103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.590049    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-systemd\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.592103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.590070    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a30f7edc-614c-48bc-9559-c7683cc316c0-kubelet-config\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:03.592103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.590093    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxjgr\" (UniqueName: \"kubernetes.io/projected/fb9d634a-bee5-4019-8a89-32065530f04c-kube-api-access-hxjgr\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.592103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.590123    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-systemd\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.592103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.590134    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a30f7edc-614c-48bc-9559-c7683cc316c0-kubelet-config\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:03.592103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.590157    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-system-cni-dir\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.592103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.590172    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdp5d\" (UniqueName: \"kubernetes.io/projected/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-kube-api-access-sdp5d\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.592103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.590202    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7725baf-5e80-4d34-9d37-43fb17f80296-lib-modules\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.592103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.590204    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7-agent-certs\") pod \"konnectivity-agent-gmfcs\" (UID: \"d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7\") " pod="kube-system/konnectivity-agent-gmfcs"
Sep 16 18:16:03.592103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.591796    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7725baf-5e80-4d34-9d37-43fb17f80296-etc-tuned\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.592103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.591830    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7725baf-5e80-4d34-9d37-43fb17f80296-tmp\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.592103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.592061    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e394908-d4ac-40f4-b8ad-232904cc2d64-ovn-node-metrics-cert\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.650559 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.650516    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wkp5\" (UniqueName: \"kubernetes.io/projected/1800b4e2-a6f1-4888-b283-fcf0ae0ac462-kube-api-access-6wkp5\") pod \"aws-ebs-csi-driver-node-z5ss4\" (UID: \"1800b4e2-a6f1-4888-b283-fcf0ae0ac462\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.650676 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.650632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmpq6\" (UniqueName: \"kubernetes.io/projected/6e394908-d4ac-40f4-b8ad-232904cc2d64-kube-api-access-bmpq6\") pod \"ovnkube-node-4wwpr\" (UID: \"6e394908-d4ac-40f4-b8ad-232904cc2d64\") " pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.653945 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.653923    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48tqw\" (UniqueName: \"kubernetes.io/projected/d93e613d-563c-4b51-93c6-4b62fb3d39aa-kube-api-access-48tqw\") pod \"network-metrics-daemon-s9p65\" (UID: \"d93e613d-563c-4b51-93c6-4b62fb3d39aa\") " pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:03.654031 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:03.653946    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:16:03.654031 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:03.653969    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:16:03.654031 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:03.653982    2562 projected.go:196] Error preparing data for projected volume kube-api-access-89l5d for pod openshift-network-diagnostics/network-check-target-rb6xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:03.654154 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:03.654046    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d podName:773c8899-dcd1-4682-9732-7761f617a2a5 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:04.154028654 +0000 UTC m=+2.018932697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-89l5d" (UniqueName: "kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d") pod "network-check-target-rb6xd" (UID: "773c8899-dcd1-4682-9732-7761f617a2a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:03.661486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.661466    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdzgc\" (UniqueName: \"kubernetes.io/projected/a7725baf-5e80-4d34-9d37-43fb17f80296-kube-api-access-bdzgc\") pod \"tuned-7czp6\" (UID: \"a7725baf-5e80-4d34-9d37-43fb17f80296\") " pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.690436 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690410    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-etc-kubernetes\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690436 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e04d0991-aae5-4412-850d-6e4685c0124a-serviceca\") pod \"node-ca-65kl9\" (UID: \"e04d0991-aae5-4412-850d-6e4685c0124a\") " pod="openshift-image-registry/node-ca-65kl9"
Sep 16 18:16:03.690666 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690455    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-cni-binary-copy\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690666 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d32eeb41-3e42-43f2-b233-339e7e76ad67-host-slash\") pod \"iptables-alerter-82fx4\" (UID: \"d32eeb41-3e42-43f2-b233-339e7e76ad67\") " pod="openshift-network-operator/iptables-alerter-82fx4"
Sep 16 18:16:03.690666 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690535    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-etc-kubernetes\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690666 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690585    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxjgr\" (UniqueName: \"kubernetes.io/projected/fb9d634a-bee5-4019-8a89-32065530f04c-kube-api-access-hxjgr\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.690666 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690599    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d32eeb41-3e42-43f2-b233-339e7e76ad67-host-slash\") pod \"iptables-alerter-82fx4\" (UID: \"d32eeb41-3e42-43f2-b233-339e7e76ad67\") " pod="openshift-network-operator/iptables-alerter-82fx4"
Sep 16 18:16:03.690666 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690622    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-system-cni-dir\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690666 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690648    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdp5d\" (UniqueName: \"kubernetes.io/projected/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-kube-api-access-sdp5d\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690671    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7-agent-certs\") pod \"konnectivity-agent-gmfcs\" (UID: \"d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7\") " pod="kube-system/konnectivity-agent-gmfcs"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690691    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-system-cni-dir\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690698    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e04d0991-aae5-4412-850d-6e4685c0124a-host\") pod \"node-ca-65kl9\" (UID: \"e04d0991-aae5-4412-850d-6e4685c0124a\") " pod="openshift-image-registry/node-ca-65kl9"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690739    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-multus-socket-dir-parent\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690744    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e04d0991-aae5-4412-850d-6e4685c0124a-host\") pod \"node-ca-65kl9\" (UID: \"e04d0991-aae5-4412-850d-6e4685c0124a\") " pod="openshift-image-registry/node-ca-65kl9"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvpf9\" (UniqueName: \"kubernetes.io/projected/d5d1838b-753e-4591-b50a-473909c62656-kube-api-access-dvpf9\") pod \"node-resolver-x5xdc\" (UID: \"d5d1838b-753e-4591-b50a-473909c62656\") " pod="openshift-dns/node-resolver-x5xdc"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb9d634a-bee5-4019-8a89-32065530f04c-cni-binary-copy\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690822    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-multus-cni-dir\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-multus-socket-dir-parent\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-run-netns\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690885    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-run-netns\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690906    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-var-lib-kubelet\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690914    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-multus-cni-dir\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690935    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-var-lib-cni-multus\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690949    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e04d0991-aae5-4412-850d-6e4685c0124a-serviceca\") pod \"node-ca-65kl9\" (UID: \"e04d0991-aae5-4412-850d-6e4685c0124a\") " pod="openshift-image-registry/node-ca-65kl9"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690960    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-multus-conf-dir\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690977    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-var-lib-kubelet\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.690985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690994    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-multus-conf-dir\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.690994    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4trzg\" (UniqueName: \"kubernetes.io/projected/d32eeb41-3e42-43f2-b233-339e7e76ad67-kube-api-access-4trzg\") pod \"iptables-alerter-82fx4\" (UID: \"d32eeb41-3e42-43f2-b233-339e7e76ad67\") " pod="openshift-network-operator/iptables-alerter-82fx4"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691032    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d5d1838b-753e-4591-b50a-473909c62656-hosts-file\") pod \"node-resolver-x5xdc\" (UID: \"d5d1838b-753e-4591-b50a-473909c62656\") " pod="openshift-dns/node-resolver-x5xdc"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691036    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-var-lib-cni-multus\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691062    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb9d634a-bee5-4019-8a89-32065530f04c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691080    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-cni-binary-copy\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691085    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d5d1838b-753e-4591-b50a-473909c62656-hosts-file\") pod \"node-resolver-x5xdc\" (UID: \"d5d1838b-753e-4591-b50a-473909c62656\") " pod="openshift-dns/node-resolver-x5xdc"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb9d634a-bee5-4019-8a89-32065530f04c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691121    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb9d634a-bee5-4019-8a89-32065530f04c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691148    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb9d634a-bee5-4019-8a89-32065530f04c-system-cni-dir\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691171    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-multus-daemon-config\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691195    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d32eeb41-3e42-43f2-b233-339e7e76ad67-iptables-alerter-script\") pod \"iptables-alerter-82fx4\" (UID: \"d32eeb41-3e42-43f2-b233-339e7e76ad67\") " pod="openshift-network-operator/iptables-alerter-82fx4"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691205    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb9d634a-bee5-4019-8a89-32065530f04c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691222    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-hostroot\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691234    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb9d634a-bee5-4019-8a89-32065530f04c-system-cni-dir\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691243    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-run-multus-certs\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691293    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-hostroot\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.691844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691293    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-cnibin\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691328    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7-konnectivity-ca\") pod \"konnectivity-agent-gmfcs\" (UID: \"d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7\") " pod="kube-system/konnectivity-agent-gmfcs"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691334    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-cnibin\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d1838b-753e-4591-b50a-473909c62656-tmp-dir\") pod \"node-resolver-x5xdc\" (UID: \"d5d1838b-753e-4591-b50a-473909c62656\") " pod="openshift-dns/node-resolver-x5xdc"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691379    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9ssl\" (UniqueName: \"kubernetes.io/projected/e04d0991-aae5-4412-850d-6e4685c0124a-kube-api-access-b9ssl\") pod \"node-ca-65kl9\" (UID: \"e04d0991-aae5-4412-850d-6e4685c0124a\") " pod="openshift-image-registry/node-ca-65kl9"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb9d634a-bee5-4019-8a89-32065530f04c-cnibin\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691426    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb9d634a-bee5-4019-8a89-32065530f04c-cni-binary-copy\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691435    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-var-lib-cni-bin\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691471    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-run-k8s-cni-cncf-io\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691477    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-var-lib-cni-bin\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691750    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb9d634a-bee5-4019-8a89-32065530f04c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691762    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-multus-daemon-config\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691762    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d32eeb41-3e42-43f2-b233-339e7e76ad67-iptables-alerter-script\") pod \"iptables-alerter-82fx4\" (UID: \"d32eeb41-3e42-43f2-b233-339e7e76ad67\") " pod="openshift-network-operator/iptables-alerter-82fx4"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691796    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-run-multus-certs\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691799    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d1838b-753e-4591-b50a-473909c62656-tmp-dir\") pod \"node-resolver-x5xdc\" (UID: \"d5d1838b-753e-4591-b50a-473909c62656\") " pod="openshift-dns/node-resolver-x5xdc"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691837    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb9d634a-bee5-4019-8a89-32065530f04c-cnibin\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.691838    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-host-run-k8s-cni-cncf-io\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.692662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.692032    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7-konnectivity-ca\") pod \"konnectivity-agent-gmfcs\" (UID: \"d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7\") " pod="kube-system/konnectivity-agent-gmfcs"
Sep 16 18:16:03.693324 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.692193    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb9d634a-bee5-4019-8a89-32065530f04c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.693420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.693402    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7-agent-certs\") pod \"konnectivity-agent-gmfcs\" (UID: \"d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7\") " pod="kube-system/konnectivity-agent-gmfcs"
Sep 16 18:16:03.703956 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.703929    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxjgr\" (UniqueName: \"kubernetes.io/projected/fb9d634a-bee5-4019-8a89-32065530f04c-kube-api-access-hxjgr\") pod \"multus-additional-cni-plugins-52wqd\" (UID: \"fb9d634a-bee5-4019-8a89-32065530f04c\") " pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.715577 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.715534    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdp5d\" (UniqueName: \"kubernetes.io/projected/5b466b4b-9e81-4fab-bfcb-5336ad5e67b6-kube-api-access-sdp5d\") pod \"multus-gws9r\" (UID: \"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6\") " pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.715756 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.715737    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9ssl\" (UniqueName: \"kubernetes.io/projected/e04d0991-aae5-4412-850d-6e4685c0124a-kube-api-access-b9ssl\") pod \"node-ca-65kl9\" (UID: \"e04d0991-aae5-4412-850d-6e4685c0124a\") " pod="openshift-image-registry/node-ca-65kl9"
Sep 16 18:16:03.716400 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.716384    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvpf9\" (UniqueName: \"kubernetes.io/projected/d5d1838b-753e-4591-b50a-473909c62656-kube-api-access-dvpf9\") pod \"node-resolver-x5xdc\" (UID: \"d5d1838b-753e-4591-b50a-473909c62656\") " pod="openshift-dns/node-resolver-x5xdc"
Sep 16 18:16:03.716515 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.716500    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4trzg\" (UniqueName: \"kubernetes.io/projected/d32eeb41-3e42-43f2-b233-339e7e76ad67-kube-api-access-4trzg\") pod \"iptables-alerter-82fx4\" (UID: \"d32eeb41-3e42-43f2-b233-339e7e76ad67\") " pod="openshift-network-operator/iptables-alerter-82fx4"
Sep 16 18:16:03.791290 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.791262    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:03.801724 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:03.801700    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e394908_d4ac_40f4_b8ad_232904cc2d64.slice/crio-c6494e3d9779b708c5c22276dce854a8919b42dc4109549f62bb26eafe77b7c3 WatchSource:0}: Error finding container c6494e3d9779b708c5c22276dce854a8919b42dc4109549f62bb26eafe77b7c3: Status 404 returned error can't find the container with id c6494e3d9779b708c5c22276dce854a8919b42dc4109549f62bb26eafe77b7c3
Sep 16 18:16:03.803170 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.803154    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4"
Sep 16 18:16:03.810353 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.810335    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7czp6"
Sep 16 18:16:03.810791 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:03.810771    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1800b4e2_a6f1_4888_b283_fcf0ae0ac462.slice/crio-1faf814b788859a99073835946b61eee6731c1cb9c1f5be12fefdeee19aeeed9 WatchSource:0}: Error finding container 1faf814b788859a99073835946b61eee6731c1cb9c1f5be12fefdeee19aeeed9: Status 404 returned error can't find the container with id 1faf814b788859a99073835946b61eee6731c1cb9c1f5be12fefdeee19aeeed9
Sep 16 18:16:03.818036 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:03.818014    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7725baf_5e80_4d34_9d37_43fb17f80296.slice/crio-7c31cc7fa77a9cbdfab4ed9f5a43cb3cc26c80969011e10fc2c82d8dbfc7af9b WatchSource:0}: Error finding container 7c31cc7fa77a9cbdfab4ed9f5a43cb3cc26c80969011e10fc2c82d8dbfc7af9b: Status 404 returned error can't find the container with id 7c31cc7fa77a9cbdfab4ed9f5a43cb3cc26c80969011e10fc2c82d8dbfc7af9b
Sep 16 18:16:03.841399 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.841377    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-52wqd"
Sep 16 18:16:03.850970 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:03.850949    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9d634a_bee5_4019_8a89_32065530f04c.slice/crio-d79b1e33c158089a8ef5a8a085c675bf4c7fa2a48190d4a261bbc1908c60b2e5 WatchSource:0}: Error finding container d79b1e33c158089a8ef5a8a085c675bf4c7fa2a48190d4a261bbc1908c60b2e5: Status 404 returned error can't find the container with id d79b1e33c158089a8ef5a8a085c675bf4c7fa2a48190d4a261bbc1908c60b2e5
Sep 16 18:16:03.861976 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.861953    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-65kl9"
Sep 16 18:16:03.868722 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.868533    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-82fx4"
Sep 16 18:16:03.869745 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:03.869720    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04d0991_aae5_4412_850d_6e4685c0124a.slice/crio-164f0f6818bab4a4726d448d3fdd667c817affab5a9e001785258c10fae82e16 WatchSource:0}: Error finding container 164f0f6818bab4a4726d448d3fdd667c817affab5a9e001785258c10fae82e16: Status 404 returned error can't find the container with id 164f0f6818bab4a4726d448d3fdd667c817affab5a9e001785258c10fae82e16
Sep 16 18:16:03.874734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.874715    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gws9r"
Sep 16 18:16:03.876610 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:03.876589    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32eeb41_3e42_43f2_b233_339e7e76ad67.slice/crio-c323c53d61cf37d4ee62f6f212cafe7b634e8a5e64f568a49dd82f165a72ad77 WatchSource:0}: Error finding container c323c53d61cf37d4ee62f6f212cafe7b634e8a5e64f568a49dd82f165a72ad77: Status 404 returned error can't find the container with id c323c53d61cf37d4ee62f6f212cafe7b634e8a5e64f568a49dd82f165a72ad77
Sep 16 18:16:03.880634 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.880514    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gmfcs"
Sep 16 18:16:03.882595 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:03.882571    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b466b4b_9e81_4fab_bfcb_5336ad5e67b6.slice/crio-4363fd01484f6b22d28fe0d81ef24d37cecde1dfc7ca037b36fb634e8138b2c8 WatchSource:0}: Error finding container 4363fd01484f6b22d28fe0d81ef24d37cecde1dfc7ca037b36fb634e8138b2c8: Status 404 returned error can't find the container with id 4363fd01484f6b22d28fe0d81ef24d37cecde1dfc7ca037b36fb634e8138b2c8
Sep 16 18:16:03.884352 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:03.884336    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x5xdc"
Sep 16 18:16:03.889850 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:03.889826    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01d46a7_a6c3_4054_8e24_0c9db3c3c9d7.slice/crio-618b7ec4accecaca7699fae295d8cd73baeb5f8994f159cd2b1b8ddb15f5cf41 WatchSource:0}: Error finding container 618b7ec4accecaca7699fae295d8cd73baeb5f8994f159cd2b1b8ddb15f5cf41: Status 404 returned error can't find the container with id 618b7ec4accecaca7699fae295d8cd73baeb5f8994f159cd2b1b8ddb15f5cf41
Sep 16 18:16:03.893490 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:03.893472    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d1838b_753e_4591_b50a_473909c62656.slice/crio-3a88764cd5f43802e095921005fe31bda0768c196f432b048d1d3711ea3a337f WatchSource:0}: Error finding container 3a88764cd5f43802e095921005fe31bda0768c196f432b048d1d3711ea3a337f: Status 404 returned error can't find the container with id 3a88764cd5f43802e095921005fe31bda0768c196f432b048d1d3711ea3a337f
Sep 16 18:16:04.094701 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.094616    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:04.094701 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.094680    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs\") pod \"network-metrics-daemon-s9p65\" (UID: \"d93e613d-563c-4b51-93c6-4b62fb3d39aa\") " pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:04.094919 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:04.094773    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:04.094919 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:04.094852    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs podName:d93e613d-563c-4b51-93c6-4b62fb3d39aa nodeName:}" failed. No retries permitted until 2026-09-16 18:16:05.094835216 +0000 UTC m=+2.959739241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs") pod "network-metrics-daemon-s9p65" (UID: "d93e613d-563c-4b51-93c6-4b62fb3d39aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:04.095026 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:04.094976    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:04.095080 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:04.095041    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret podName:a30f7edc-614c-48bc-9559-c7683cc316c0 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:05.095024262 +0000 UTC m=+2.959928299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret") pod "global-pull-secret-syncer-ks9dh" (UID: "a30f7edc-614c-48bc-9559-c7683cc316c0") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:04.195225 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.195180    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89l5d\" (UniqueName: \"kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d\") pod \"network-check-target-rb6xd\" (UID: \"773c8899-dcd1-4682-9732-7761f617a2a5\") " pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:04.195407 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:04.195357    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:16:04.195407 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:04.195380    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:16:04.195407 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:04.195390    2562 projected.go:196] Error preparing data for projected volume kube-api-access-89l5d for pod openshift-network-diagnostics/network-check-target-rb6xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:04.195588 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:04.195442    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d podName:773c8899-dcd1-4682-9732-7761f617a2a5 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:05.195424381 +0000 UTC m=+3.060328417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-89l5d" (UniqueName: "kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d") pod "network-check-target-rb6xd" (UID: "773c8899-dcd1-4682-9732-7761f617a2a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:04.536907 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.536838    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:11:03 +0000 UTC" deadline="2028-04-11 18:41:31.84078572 +0000 UTC"
Sep 16 18:16:04.536907 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.536905    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13752h25m27.303885211s"
Sep 16 18:16:04.566740 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.566681    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gmfcs" event={"ID":"d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7","Type":"ContainerStarted","Data":"618b7ec4accecaca7699fae295d8cd73baeb5f8994f159cd2b1b8ddb15f5cf41"}
Sep 16 18:16:04.577882 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.577822    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-82fx4" event={"ID":"d32eeb41-3e42-43f2-b233-339e7e76ad67","Type":"ContainerStarted","Data":"c323c53d61cf37d4ee62f6f212cafe7b634e8a5e64f568a49dd82f165a72ad77"}
Sep 16 18:16:04.583095 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.583051    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wqd" event={"ID":"fb9d634a-bee5-4019-8a89-32065530f04c","Type":"ContainerStarted","Data":"d79b1e33c158089a8ef5a8a085c675bf4c7fa2a48190d4a261bbc1908c60b2e5"}
Sep 16 18:16:04.595564 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.595484    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-48.ec2.internal" event={"ID":"f512f1af51c00a31a72cfdbb64042241","Type":"ContainerStarted","Data":"c2d75e9dcd6c50b240e0386991755801b13c02a4a5c04ea27a147294d1f7878f"}
Sep 16 18:16:04.602213 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.602176    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x5xdc" event={"ID":"d5d1838b-753e-4591-b50a-473909c62656","Type":"ContainerStarted","Data":"3a88764cd5f43802e095921005fe31bda0768c196f432b048d1d3711ea3a337f"}
Sep 16 18:16:04.614618 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.611360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gws9r" event={"ID":"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6","Type":"ContainerStarted","Data":"4363fd01484f6b22d28fe0d81ef24d37cecde1dfc7ca037b36fb634e8138b2c8"}
Sep 16 18:16:04.623752 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.623678    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-65kl9" event={"ID":"e04d0991-aae5-4412-850d-6e4685c0124a","Type":"ContainerStarted","Data":"164f0f6818bab4a4726d448d3fdd667c817affab5a9e001785258c10fae82e16"}
Sep 16 18:16:04.632223 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.632192    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7czp6" event={"ID":"a7725baf-5e80-4d34-9d37-43fb17f80296","Type":"ContainerStarted","Data":"7c31cc7fa77a9cbdfab4ed9f5a43cb3cc26c80969011e10fc2c82d8dbfc7af9b"}
Sep 16 18:16:04.634857 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.634827    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4" event={"ID":"1800b4e2-a6f1-4888-b283-fcf0ae0ac462","Type":"ContainerStarted","Data":"1faf814b788859a99073835946b61eee6731c1cb9c1f5be12fefdeee19aeeed9"}
Sep 16 18:16:04.637781 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.637747    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr" event={"ID":"6e394908-d4ac-40f4-b8ad-232904cc2d64","Type":"ContainerStarted","Data":"c6494e3d9779b708c5c22276dce854a8919b42dc4109549f62bb26eafe77b7c3"}
Sep 16 18:16:04.640118 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.640093    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal" event={"ID":"5adff8773b3a030a2e2de8635bd6ee73","Type":"ContainerStarted","Data":"d7d3ff30fe46083977f058ec59c469c3bc724cd1938fdeaf465d9a4b1c9ab68a"}
Sep 16 18:16:04.920107 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:04.919861    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:16:05.102848 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:05.102810    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs\") pod \"network-metrics-daemon-s9p65\" (UID: \"d93e613d-563c-4b51-93c6-4b62fb3d39aa\") " pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:05.103051 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:05.102897    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:05.103051 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:05.103017    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:05.103167 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:05.103076    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret podName:a30f7edc-614c-48bc-9559-c7683cc316c0 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:07.103059542 +0000 UTC m=+4.967963573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret") pod "global-pull-secret-syncer-ks9dh" (UID: "a30f7edc-614c-48bc-9559-c7683cc316c0") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:05.103468 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:05.103450    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:05.103573 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:05.103505    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs podName:d93e613d-563c-4b51-93c6-4b62fb3d39aa nodeName:}" failed. No retries permitted until 2026-09-16 18:16:07.103489794 +0000 UTC m=+4.968393821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs") pod "network-metrics-daemon-s9p65" (UID: "d93e613d-563c-4b51-93c6-4b62fb3d39aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:05.203384 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:05.203352    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89l5d\" (UniqueName: \"kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d\") pod \"network-check-target-rb6xd\" (UID: \"773c8899-dcd1-4682-9732-7761f617a2a5\") " pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:05.203587 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:05.203501    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:16:05.203587 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:05.203519    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:16:05.203587 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:05.203531    2562 projected.go:196] Error preparing data for projected volume kube-api-access-89l5d for pod openshift-network-diagnostics/network-check-target-rb6xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:05.203755 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:05.203605    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d podName:773c8899-dcd1-4682-9732-7761f617a2a5 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:07.203588223 +0000 UTC m=+5.068492273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-89l5d" (UniqueName: "kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d") pod "network-check-target-rb6xd" (UID: "773c8899-dcd1-4682-9732-7761f617a2a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:05.245779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:05.245750    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:16:05.537883 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:05.537789    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:11:03 +0000 UTC" deadline="2028-06-24 18:18:45.237619528 +0000 UTC"
Sep 16 18:16:05.537883 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:05.537828    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15528h2m39.699797036s"
Sep 16 18:16:05.544204 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:05.544180    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:05.544341 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:05.544304    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:05.544727 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:05.544708    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:05.544821 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:05.544803    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:05.544884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:05.544873    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:05.544974 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:05.544957    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:06.493313 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:06.493101    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:16:07.120755 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:07.120717    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs\") pod \"network-metrics-daemon-s9p65\" (UID: \"d93e613d-563c-4b51-93c6-4b62fb3d39aa\") " pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:07.121187 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:07.120808    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:07.121187 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:07.120950    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:07.121187 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:07.121012    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret podName:a30f7edc-614c-48bc-9559-c7683cc316c0 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:11.120994478 +0000 UTC m=+8.985898526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret") pod "global-pull-secret-syncer-ks9dh" (UID: "a30f7edc-614c-48bc-9559-c7683cc316c0") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:07.121409 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:07.121390    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:07.121460 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:07.121447    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs podName:d93e613d-563c-4b51-93c6-4b62fb3d39aa nodeName:}" failed. No retries permitted until 2026-09-16 18:16:11.121431218 +0000 UTC m=+8.986335246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs") pod "network-metrics-daemon-s9p65" (UID: "d93e613d-563c-4b51-93c6-4b62fb3d39aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:07.221828 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:07.221793    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89l5d\" (UniqueName: \"kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d\") pod \"network-check-target-rb6xd\" (UID: \"773c8899-dcd1-4682-9732-7761f617a2a5\") " pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:07.222024 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:07.222007    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:16:07.222081 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:07.222036    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:16:07.222081 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:07.222050    2562 projected.go:196] Error preparing data for projected volume kube-api-access-89l5d for pod openshift-network-diagnostics/network-check-target-rb6xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:07.222176 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:07.222108    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d podName:773c8899-dcd1-4682-9732-7761f617a2a5 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:11.222090231 +0000 UTC m=+9.086994261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-89l5d" (UniqueName: "kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d") pod "network-check-target-rb6xd" (UID: "773c8899-dcd1-4682-9732-7761f617a2a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:07.543828 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:07.543793    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:07.544015 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:07.543938    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:07.544015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:07.543978    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:07.544139 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:07.544101    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:07.544197 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:07.544154    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:07.544247 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:07.544222    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:09.544258 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:09.544222    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:09.544855 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:09.544320    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:09.544855 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:09.544364    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:09.544855 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:09.544429    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:09.544855 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:09.544233    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:09.544855 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:09.544516    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:11.156454 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:11.156407    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs\") pod \"network-metrics-daemon-s9p65\" (UID: \"d93e613d-563c-4b51-93c6-4b62fb3d39aa\") " pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:11.156941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:11.156506    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:11.156941 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:11.156637    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:11.156941 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:11.156643    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:11.156941 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:11.156699    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret podName:a30f7edc-614c-48bc-9559-c7683cc316c0 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:19.156680187 +0000 UTC m=+17.021584216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret") pod "global-pull-secret-syncer-ks9dh" (UID: "a30f7edc-614c-48bc-9559-c7683cc316c0") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:11.156941 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:11.156722    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs podName:d93e613d-563c-4b51-93c6-4b62fb3d39aa nodeName:}" failed. No retries permitted until 2026-09-16 18:16:19.15670998 +0000 UTC m=+17.021614010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs") pod "network-metrics-daemon-s9p65" (UID: "d93e613d-563c-4b51-93c6-4b62fb3d39aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:11.257444 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:11.257409    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89l5d\" (UniqueName: \"kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d\") pod \"network-check-target-rb6xd\" (UID: \"773c8899-dcd1-4682-9732-7761f617a2a5\") " pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:11.257650 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:11.257628    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:16:11.257736 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:11.257659    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:16:11.257736 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:11.257673    2562 projected.go:196] Error preparing data for projected volume kube-api-access-89l5d for pod openshift-network-diagnostics/network-check-target-rb6xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:11.257829 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:11.257740    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d podName:773c8899-dcd1-4682-9732-7761f617a2a5 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:19.257721372 +0000 UTC m=+17.122625401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-89l5d" (UniqueName: "kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d") pod "network-check-target-rb6xd" (UID: "773c8899-dcd1-4682-9732-7761f617a2a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:11.544306 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:11.543640    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:11.544306 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:11.543774    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:11.544306 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:11.544156    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:11.544306 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:11.544265    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:11.544689 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:11.544347    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:11.544689 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:11.544457    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:13.544321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:13.544288    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:13.544905 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:13.544401    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:13.544905 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:13.544406    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:13.544905 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:13.544415    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:13.544905 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:13.544494    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:13.544905 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:13.544594    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:15.544042 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:15.544006    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:15.544515 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:15.544124    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:15.544515 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:15.544141    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:15.544515 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:15.544164    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:15.544515 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:15.544280    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:15.544515 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:15.544351    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:17.544390 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:17.544355    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:17.544838 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:17.544360    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:17.544838 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:17.544476    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:17.544838 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:17.544367    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:17.544838 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:17.544556    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:17.544838 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:17.544644    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:19.218704 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:19.218672    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:19.219319 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:19.218715    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs\") pod \"network-metrics-daemon-s9p65\" (UID: \"d93e613d-563c-4b51-93c6-4b62fb3d39aa\") " pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:19.219319 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:19.218824    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:19.219319 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:19.218877    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs podName:d93e613d-563c-4b51-93c6-4b62fb3d39aa nodeName:}" failed. No retries permitted until 2026-09-16 18:16:35.218863127 +0000 UTC m=+33.083767153 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs") pod "network-metrics-daemon-s9p65" (UID: "d93e613d-563c-4b51-93c6-4b62fb3d39aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:19.219319 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:19.218824    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:19.219319 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:19.218918    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret podName:a30f7edc-614c-48bc-9559-c7683cc316c0 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:35.218911071 +0000 UTC m=+33.083815112 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret") pod "global-pull-secret-syncer-ks9dh" (UID: "a30f7edc-614c-48bc-9559-c7683cc316c0") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:19.319170 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:19.319133    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89l5d\" (UniqueName: \"kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d\") pod \"network-check-target-rb6xd\" (UID: \"773c8899-dcd1-4682-9732-7761f617a2a5\") " pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:19.319405 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:19.319337    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:16:19.319405 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:19.319363    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:16:19.319405 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:19.319376    2562 projected.go:196] Error preparing data for projected volume kube-api-access-89l5d for pod openshift-network-diagnostics/network-check-target-rb6xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:19.319660 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:19.319440    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d podName:773c8899-dcd1-4682-9732-7761f617a2a5 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:35.319422928 +0000 UTC m=+33.184326954 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-89l5d" (UniqueName: "kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d") pod "network-check-target-rb6xd" (UID: "773c8899-dcd1-4682-9732-7761f617a2a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:19.544220 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:19.544124    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:19.544390 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:19.544132    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:19.544390 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:19.544261    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:19.544390 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:19.544254    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:19.544390 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:19.544341    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:19.544577 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:19.544413    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:21.543855 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:21.543820    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:21.543855 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:21.543853    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:21.544368 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:21.543825    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:21.544368 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:21.543972    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:21.544368 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:21.544075    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:21.544368 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:21.544192    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:23.544506 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.544065    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:23.545605 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:23.544641    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:23.545605 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.544736    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:23.545605 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:23.544802    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:23.545605 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.544858    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:23.545605 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:23.544932    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:23.686360 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.686320    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-65kl9" event={"ID":"e04d0991-aae5-4412-850d-6e4685c0124a","Type":"ContainerStarted","Data":"a1c0847900ee3123e88bd42fd5c9e948e69d79bbfe0e48e6a63ebc46984668cb"}
Sep 16 18:16:23.688037 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.688007    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7czp6" event={"ID":"a7725baf-5e80-4d34-9d37-43fb17f80296","Type":"ContainerStarted","Data":"e09eb25ad0441e414cd2f2869357bd0c0f0900543430c4c6ba16dff2ab07e9bc"}
Sep 16 18:16:23.689507 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.689477    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4" event={"ID":"1800b4e2-a6f1-4888-b283-fcf0ae0ac462","Type":"ContainerStarted","Data":"93a9c00aab799a23ae453db1cf9f42a238c9b84a5b889742223881ec460c45e5"}
Sep 16 18:16:23.692458 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.692437    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:16:23.692859 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.692840    2562 generic.go:356] "Generic (PLEG): container finished" podID="6e394908-d4ac-40f4-b8ad-232904cc2d64" containerID="2183580d3cb7033561b8ac89fe08fd942287c8084a9972561b2f35747023982b" exitCode=1
Sep 16 18:16:23.692928 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.692904    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr" event={"ID":"6e394908-d4ac-40f4-b8ad-232904cc2d64","Type":"ContainerStarted","Data":"1144b48a8950f861677d9890217e203cbf48238fcb79c1b2b7aea972c287920b"}
Sep 16 18:16:23.692984 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.692968    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr" event={"ID":"6e394908-d4ac-40f4-b8ad-232904cc2d64","Type":"ContainerStarted","Data":"6d2f59b5bb456ce6aca46a18c6707413dfbdb66de20f9d63b4c4839ed8547229"}
Sep 16 18:16:23.693025 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.693013    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr" event={"ID":"6e394908-d4ac-40f4-b8ad-232904cc2d64","Type":"ContainerStarted","Data":"02dbaf86d68203e59581bd1525738c78a2bc7a50f4a5cb019c5be5bc7dd71218"}
Sep 16 18:16:23.693071 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.693059    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr" event={"ID":"6e394908-d4ac-40f4-b8ad-232904cc2d64","Type":"ContainerStarted","Data":"a52211ad8b6a950a44c45a42a25f5b6d5100398c859aa42d5a3d2afbdbdb0412"}
Sep 16 18:16:23.693114 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.693103    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr" event={"ID":"6e394908-d4ac-40f4-b8ad-232904cc2d64","Type":"ContainerDied","Data":"2183580d3cb7033561b8ac89fe08fd942287c8084a9972561b2f35747023982b"}
Sep 16 18:16:23.693172 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.693149    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr" event={"ID":"6e394908-d4ac-40f4-b8ad-232904cc2d64","Type":"ContainerStarted","Data":"61b9903050e9fe79f7d71e786b9dfb8ce256af149a43aab338e0ab9dd265744b"}
Sep 16 18:16:23.694365 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.694345    2562 generic.go:356] "Generic (PLEG): container finished" podID="5adff8773b3a030a2e2de8635bd6ee73" containerID="d54f9b75b39a9bac2dcc203232cc87f53bb5b04efa10a4b54dc3dde15547adf8" exitCode=0
Sep 16 18:16:23.694455 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.694404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal" event={"ID":"5adff8773b3a030a2e2de8635bd6ee73","Type":"ContainerDied","Data":"d54f9b75b39a9bac2dcc203232cc87f53bb5b04efa10a4b54dc3dde15547adf8"}
Sep 16 18:16:23.695708 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.695687    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gmfcs" event={"ID":"d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7","Type":"ContainerStarted","Data":"d9baa85b85fe090d428691b7ab246d0614815df5b1163a85c7ad1abafdd38cf6"}
Sep 16 18:16:23.697223 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.697200    2562 generic.go:356] "Generic (PLEG): container finished" podID="fb9d634a-bee5-4019-8a89-32065530f04c" containerID="9bcc898bd2539f31a415ba70f547b313474261fd00f0f9355c7dc43e5ff48c4b" exitCode=0
Sep 16 18:16:23.697314 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.697272    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wqd" event={"ID":"fb9d634a-bee5-4019-8a89-32065530f04c","Type":"ContainerDied","Data":"9bcc898bd2539f31a415ba70f547b313474261fd00f0f9355c7dc43e5ff48c4b"}
Sep 16 18:16:23.700665 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.700637    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-48.ec2.internal" event={"ID":"f512f1af51c00a31a72cfdbb64042241","Type":"ContainerStarted","Data":"0f8d56f25500230c60a65d422bbe48375f4e4177a4fe5479bf3cae21981eb941"}
Sep 16 18:16:23.702362 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.702334    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x5xdc" event={"ID":"d5d1838b-753e-4591-b50a-473909c62656","Type":"ContainerStarted","Data":"774026243d3cb5f9dfb9ca051934917503b9e2359c2070ab10338fb48f50fcca"}
Sep 16 18:16:23.703884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.703864    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gws9r" event={"ID":"5b466b4b-9e81-4fab-bfcb-5336ad5e67b6","Type":"ContainerStarted","Data":"f6fd55ae4826d59eed2d4b4985c5dbd1bb9dbc69fc024d17fc1d03102bbcdeaa"}
Sep 16 18:16:23.704226 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.704184    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-65kl9" podStartSLOduration=6.852318501 podStartE2EDuration="21.704170788s" totalImagesPullingTime="14.851852287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:23.703538637 +0000 UTC m=+21.568442684" watchObservedRunningTime="2026-09-16 18:16:23.704170788 +0000 UTC m=+21.569074835"
Sep 16 18:16:23.749280 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.749161    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-x5xdc" podStartSLOduration=2.249255296 podStartE2EDuration="20.749137872s" totalImagesPullingTime="18.499882576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:23.748736583 +0000 UTC m=+21.613640630" watchObservedRunningTime="2026-09-16 18:16:23.749137872 +0000 UTC m=+21.614041922"
Sep 16 18:16:23.764835 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.764785    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-48.ec2.internal" podStartSLOduration=20.764769884 podStartE2EDuration="20.764769884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:23.764419498 +0000 UTC m=+21.629323542" watchObservedRunningTime="2026-09-16 18:16:23.764769884 +0000 UTC m=+21.629673931"
Sep 16 18:16:23.784372 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.784324    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7czp6" podStartSLOduration=3.155900982 podStartE2EDuration="21.784308105s" totalImagesPullingTime="18.628407123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:23.783198399 +0000 UTC m=+21.648102447" watchObservedRunningTime="2026-09-16 18:16:23.784308105 +0000 UTC m=+21.649212154"
Sep 16 18:16:23.799637 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.799584    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gmfcs" podStartSLOduration=3.295684566 podStartE2EDuration="21.799567482s" totalImagesPullingTime="18.503882916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:23.798984699 +0000 UTC m=+21.663888746" watchObservedRunningTime="2026-09-16 18:16:23.799567482 +0000 UTC m=+21.664471521"
Sep 16 18:16:23.824862 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:23.824816    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gws9r" podStartSLOduration=2.444133812 podStartE2EDuration="21.824800198s" totalImagesPullingTime="19.380666386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:23.823575461 +0000 UTC m=+21.688479506" watchObservedRunningTime="2026-09-16 18:16:23.824800198 +0000 UTC m=+21.689704245"
Sep 16 18:16:24.711050 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:24.711008    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-82fx4" event={"ID":"d32eeb41-3e42-43f2-b233-339e7e76ad67","Type":"ContainerStarted","Data":"ca82fe5d3b102eff861abd001596bd8895a0f425a59449d4bd1dc9a4a6afa082"}
Sep 16 18:16:24.728403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:24.728346    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-82fx4" podStartSLOduration=4.161336977 podStartE2EDuration="22.728325581s" totalImagesPullingTime="18.566988604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:24.7275612 +0000 UTC m=+22.592465240" watchObservedRunningTime="2026-09-16 18:16:24.728325581 +0000 UTC m=+22.593229632"
Sep 16 18:16:25.031246 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:25.031212    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:16:25.544597 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:25.544399    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:25.544819 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:25.544436    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:25.544819 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:25.544674    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:25.544819 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:25.544759    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:25.544819 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:25.544436    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:25.545049 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:25.544850    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:25.546491 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:25.546402    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:16:25.031237523Z","UUID":"fc42a921-c314-4435-8de1-b75564cd296c","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:16:25.548496 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:25.548473    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:16:25.548725 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:25.548513    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:16:25.715593 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:25.715558    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4" event={"ID":"1800b4e2-a6f1-4888-b283-fcf0ae0ac462","Type":"ContainerStarted","Data":"5679bb93851ab56c75d4297548f972cc601f6019a89810436430826a2430f752"}
Sep 16 18:16:25.719223 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:25.719194    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:16:25.719689 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:25.719659    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr" event={"ID":"6e394908-d4ac-40f4-b8ad-232904cc2d64","Type":"ContainerStarted","Data":"296ec3ea98121a9c11a700d538a1a86aea90d226be176c7a97d53e1a82633f7f"}
Sep 16 18:16:25.721703 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:25.721676    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal" event={"ID":"5adff8773b3a030a2e2de8635bd6ee73","Type":"ContainerStarted","Data":"751bccec883929cef1ab946b137661668705fce357aa289a6c6e24b54d6d18f4"}
Sep 16 18:16:26.291284 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:26.291246    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gmfcs"
Sep 16 18:16:26.292168 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:26.292141    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gmfcs"
Sep 16 18:16:26.313366 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:26.313316    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-48.ec2.internal" podStartSLOduration=23.31329664 podStartE2EDuration="23.31329664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:25.751084599 +0000 UTC m=+23.615988648" watchObservedRunningTime="2026-09-16 18:16:26.31329664 +0000 UTC m=+24.178200684"
Sep 16 18:16:27.544006 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:27.543971    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:27.544493 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:27.544086    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:27.544493 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:27.544094    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:27.544493 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:27.544150    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:27.544493 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:27.544245    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:27.544493 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:27.544347    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:28.732273 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:28.732112    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:16:28.732901 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:28.732657    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr" event={"ID":"6e394908-d4ac-40f4-b8ad-232904cc2d64","Type":"ContainerStarted","Data":"1ba3a10b56fec76f7867a2f2e258a594a47afcd742af8e190a4afef27878a46d"}
Sep 16 18:16:28.732983 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:28.732953    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:28.733141 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:28.733125    2562 scope.go:117] "RemoveContainer" containerID="2183580d3cb7033561b8ac89fe08fd942287c8084a9972561b2f35747023982b"
Sep 16 18:16:28.734770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:28.734741    2562 generic.go:356] "Generic (PLEG): container finished" podID="fb9d634a-bee5-4019-8a89-32065530f04c" containerID="3fef84d6231ff990819632d4c9c42d55d40c94d9c0e715e43cff7b053bc1e5ea" exitCode=0
Sep 16 18:16:28.734864 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:28.734771    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wqd" event={"ID":"fb9d634a-bee5-4019-8a89-32065530f04c","Type":"ContainerDied","Data":"3fef84d6231ff990819632d4c9c42d55d40c94d9c0e715e43cff7b053bc1e5ea"}
Sep 16 18:16:28.737374 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:28.737348    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4" event={"ID":"1800b4e2-a6f1-4888-b283-fcf0ae0ac462","Type":"ContainerStarted","Data":"b4ecfe0b2dfb3c23f389a829dc5d33eae2d505c415c2cacd7881e960a5196022"}
Sep 16 18:16:28.749753 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:28.749725    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:28.876947 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:28.876891    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5ss4" podStartSLOduration=3.877384619 podStartE2EDuration="26.876872199s" totalImagesPullingTime="22.99948758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:28.876719718 +0000 UTC m=+26.741623766" watchObservedRunningTime="2026-09-16 18:16:28.876872199 +0000 UTC m=+26.741776275"
Sep 16 18:16:29.544473 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:29.544414    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:29.544754 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:29.544564    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:29.544922 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:29.544885    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:29.544922 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:29.544903    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:29.545101 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:29.545002    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:29.545158 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:29.545105    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:29.743666 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:29.743639    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:16:29.744255 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:29.744181    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr" event={"ID":"6e394908-d4ac-40f4-b8ad-232904cc2d64","Type":"ContainerStarted","Data":"76801f0b4a0c268d3ff9a0f9268faa88787c6cdc19304f9c689115434e29d16f"}
Sep 16 18:16:29.744826 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:29.744723    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:29.744963 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:29.744887    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:29.760198 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:29.760172    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:16:29.828671 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:29.828579    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr" podStartSLOduration=8.897727604 podStartE2EDuration="27.828562601s" totalImagesPullingTime="18.930834997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:29.785273966 +0000 UTC m=+27.650178027" watchObservedRunningTime="2026-09-16 18:16:29.828562601 +0000 UTC m=+27.693466679"
Sep 16 18:16:30.261515 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:30.261474    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s9p65"]
Sep 16 18:16:30.261686 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:30.261627    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:30.261774 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:30.261751    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:30.268416 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:30.268389    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ks9dh"]
Sep 16 18:16:30.268536 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:30.268476    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:30.268629 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:30.268574    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:30.273631 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:30.273592    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rb6xd"]
Sep 16 18:16:30.273738 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:30.273687    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:30.273810 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:30.273789    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:30.748795 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:30.748760    2562 generic.go:356] "Generic (PLEG): container finished" podID="fb9d634a-bee5-4019-8a89-32065530f04c" containerID="3bac61044b5b1cad8bd667323d7f3837feafd623f1727fbc0664225ec195e28a" exitCode=0
Sep 16 18:16:30.749327 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:30.748834    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wqd" event={"ID":"fb9d634a-bee5-4019-8a89-32065530f04c","Type":"ContainerDied","Data":"3bac61044b5b1cad8bd667323d7f3837feafd623f1727fbc0664225ec195e28a"}
Sep 16 18:16:31.543737 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:31.543645    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:31.543737 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:31.543646    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:31.543920 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:31.543766    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:31.543920 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:31.543799    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:32.544621 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:32.544253    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:32.544621 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:32.544611    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:32.755750 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:32.755717    2562 generic.go:356] "Generic (PLEG): container finished" podID="fb9d634a-bee5-4019-8a89-32065530f04c" containerID="dcc17b08cd5a5af2ac7b27e6f91f2fc0ebc4c0e833dcbbe4b0d7d29f5d03f64a" exitCode=0
Sep 16 18:16:32.755952 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:32.755770    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wqd" event={"ID":"fb9d634a-bee5-4019-8a89-32065530f04c","Type":"ContainerDied","Data":"dcc17b08cd5a5af2ac7b27e6f91f2fc0ebc4c0e833dcbbe4b0d7d29f5d03f64a"}
Sep 16 18:16:33.544091 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:33.544058    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:33.544280 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:33.544174    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:33.544280 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:33.544242    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:33.544389 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:33.544368    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:33.856196 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:33.856094    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gmfcs"
Sep 16 18:16:33.856639 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:33.856227    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:16:33.856860 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:33.856831    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gmfcs"
Sep 16 18:16:34.543533 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:34.543456    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:34.543708 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:34.543601    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9p65" podUID="d93e613d-563c-4b51-93c6-4b62fb3d39aa"
Sep 16 18:16:35.236240 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.236197    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs\") pod \"network-metrics-daemon-s9p65\" (UID: \"d93e613d-563c-4b51-93c6-4b62fb3d39aa\") " pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:35.236828 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.236291    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:35.236828 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:35.236368    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:35.236828 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:35.236425    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:35.236828 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:35.236448    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs podName:d93e613d-563c-4b51-93c6-4b62fb3d39aa nodeName:}" failed. No retries permitted until 2026-09-16 18:17:07.236422927 +0000 UTC m=+65.101326953 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs") pod "network-metrics-daemon-s9p65" (UID: "d93e613d-563c-4b51-93c6-4b62fb3d39aa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:16:35.236828 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:35.236477    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret podName:a30f7edc-614c-48bc-9559-c7683cc316c0 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:07.236462444 +0000 UTC m=+65.101366474 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret") pod "global-pull-secret-syncer-ks9dh" (UID: "a30f7edc-614c-48bc-9559-c7683cc316c0") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:16:35.337141 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.337099    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89l5d\" (UniqueName: \"kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d\") pod \"network-check-target-rb6xd\" (UID: \"773c8899-dcd1-4682-9732-7761f617a2a5\") " pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:35.337336 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:35.337259    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:16:35.337336 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:35.337288    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:16:35.337336 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:35.337301    2562 projected.go:196] Error preparing data for projected volume kube-api-access-89l5d for pod openshift-network-diagnostics/network-check-target-rb6xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:35.337509 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:35.337372    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d podName:773c8899-dcd1-4682-9732-7761f617a2a5 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:07.337351052 +0000 UTC m=+65.202255098 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-89l5d" (UniqueName: "kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d") pod "network-check-target-rb6xd" (UID: "773c8899-dcd1-4682-9732-7761f617a2a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:16:35.544510 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.544424    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:35.544510 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.544440    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:35.544770 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:35.544573    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rb6xd" podUID="773c8899-dcd1-4682-9732-7761f617a2a5"
Sep 16 18:16:35.544770 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:35.544697    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks9dh" podUID="a30f7edc-614c-48bc-9559-c7683cc316c0"
Sep 16 18:16:35.925707 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.925623    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-48.ec2.internal" event="NodeReady"
Sep 16 18:16:35.925881 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.925803    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:16:35.973354 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.973316    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84f6855565-qv2zc"]
Sep 16 18:16:35.975460 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.975426    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"]
Sep 16 18:16:35.975698 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.975675    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:35.977338 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.977314    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-c65ff548d-bxgzt"]
Sep 16 18:16:35.977482 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.977396    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"
Sep 16 18:16:35.978993 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.978974    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcvrf"]
Sep 16 18:16:35.979106 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.979087    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:35.979677 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.979655    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:16:35.979777 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.979701    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:16:35.979777 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.979719    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8b29l\""
Sep 16 18:16:35.980104 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.980092    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:16:35.980691 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.980678    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-wnqkf"]
Sep 16 18:16:35.980807 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.980792    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcvrf"
Sep 16 18:16:35.985533 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.985333    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"]
Sep 16 18:16:35.986972 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.986390    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:35.986972 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.986648    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nsl9n\""
Sep 16 18:16:35.987621 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.987605    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 18:16:35.987793 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.987777    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:16:35.987793 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.987789    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:16:35.987919 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.987832    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 18:16:35.987919 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.987835    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 18:16:35.987919 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.987789    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 18:16:35.988221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.988187    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 18:16:35.988221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.988217    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:16:35.988336 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.988316    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gkkf5\""
Sep 16 18:16:35.988417 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.988399    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:16:35.988606 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.988587    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:16:35.989058 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.989041    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-54s96\""
Sep 16 18:16:35.989836 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.989510    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 18:16:35.990567 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.990527    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-mxm76"]
Sep 16 18:16:35.990876 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.990854    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 18:16:35.991801 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.991783    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:16:35.992603 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.992589    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:35.995147 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.995130    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-h76sd\""
Sep 16 18:16:35.995320 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.995307    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:16:35.996137 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.996115    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:16:35.996137 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.996128    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7w9c7\""
Sep 16 18:16:35.996487 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.996474    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 18:16:35.996714 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.996694    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 18:16:35.996794 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.996728    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:16:35.998519 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.998503    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 18:16:35.998519 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.998510    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 18:16:35.998684 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.998531    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:16:35.999707 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.999689    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 18:16:35.999707 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.999697    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 18:16:35.999707 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.999697    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-h5895\""
Sep 16 18:16:35.999899 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:35.999702    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:16:36.000538 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.000478    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 18:16:36.002426 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.002402    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"]
Sep 16 18:16:36.007479 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.007456    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 18:16:36.008820 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.008798    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 18:16:36.013649 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.013626    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcvrf"]
Sep 16 18:16:36.014941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.014922    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-wnqkf"]
Sep 16 18:16:36.016555 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.016491    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"]
Sep 16 18:16:36.018405 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.017771    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84f6855565-qv2zc"]
Sep 16 18:16:36.020169 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.019863    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bcg6p"]
Sep 16 18:16:36.028169 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.028127    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vnc87"]
Sep 16 18:16:36.028312 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.028246    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:36.030740 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.030703    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-mxm76"]
Sep 16 18:16:36.030865 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.030852    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vnc87"
Sep 16 18:16:36.031443 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.031373    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:16:36.031443 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.031375    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:16:36.031680 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.031484    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pfff7\""
Sep 16 18:16:36.035512 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.035389    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:16:36.035852 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.035557    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:16:36.035852 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.035599    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-76n2b\""
Sep 16 18:16:36.035852 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.035685    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:16:36.037689 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.037518    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bcg6p"]
Sep 16 18:16:36.041019 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.040923    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c65ff548d-bxgzt"]
Sep 16 18:16:36.041482 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.041288    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vnc87"]
Sep 16 18:16:36.041824 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.041791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da984918-11f3-4d23-9381-bd3cc174771f-trusted-ca\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.041900 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.041843    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da984918-11f3-4d23-9381-bd3cc174771f-registry-certificates\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.041900 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.041877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:36.041983 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.041903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzk8v\" (UniqueName: \"kubernetes.io/projected/92d1ca76-661a-47d9-bb84-e11ee89825e7-kube-api-access-tzk8v\") pod \"volume-data-source-validator-86695b7cc7-gcvrf\" (UID: \"92d1ca76-661a-47d9-bb84-e11ee89825e7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcvrf"
Sep 16 18:16:36.041983 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.041932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-trusted-ca-bundle\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.041983 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.041958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qxg8\" (UniqueName: \"kubernetes.io/projected/67a74600-f466-4ae4-b4bf-67ab02745509-kube-api-access-4qxg8\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:36.042112 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.041990    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da984918-11f3-4d23-9381-bd3cc174771f-image-registry-private-configuration\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.042112 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042015    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.042112 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh248\" (UniqueName: \"kubernetes.io/projected/d3828f83-14bf-4688-ba93-b65e204a4118-kube-api-access-bh248\") pod \"cluster-samples-operator-86499c89c4-mjv99\" (UID: \"d3828f83-14bf-4688-ba93-b65e204a4118\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"
Sep 16 18:16:36.042269 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042158    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-frd2q\" (UID: \"06266e44-9985-4fdc-af4f-1476cabba0fd\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:16:36.042269 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042189    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-snapshots\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.042269 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042216    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9b403df-9c62-4d05-b8cc-2d2e3d94f521-config\") pod \"console-operator-696884fb9c-mxm76\" (UID: \"e9b403df-9c62-4d05-b8cc-2d2e3d94f521\") " pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:36.042269 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9b403df-9c62-4d05-b8cc-2d2e3d94f521-trusted-ca\") pod \"console-operator-696884fb9c-mxm76\" (UID: \"e9b403df-9c62-4d05-b8cc-2d2e3d94f521\") " pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:36.042269 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042265    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06266e44-9985-4fdc-af4f-1476cabba0fd-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-frd2q\" (UID: \"06266e44-9985-4fdc-af4f-1476cabba0fd\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:16:36.042511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042289    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-bound-sa-token\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.042511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042313    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrwgb\" (UniqueName: \"kubernetes.io/projected/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-kube-api-access-hrwgb\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.042511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042341    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9b403df-9c62-4d05-b8cc-2d2e3d94f521-serving-cert\") pod \"console-operator-696884fb9c-mxm76\" (UID: \"e9b403df-9c62-4d05-b8cc-2d2e3d94f521\") " pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:36.042511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042365    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d69kn\" (UniqueName: \"kubernetes.io/projected/06266e44-9985-4fdc-af4f-1476cabba0fd-kube-api-access-d69kn\") pod \"cluster-monitoring-operator-6cd954b897-frd2q\" (UID: \"06266e44-9985-4fdc-af4f-1476cabba0fd\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:16:36.042511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-tmp\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.042511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042412    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-service-ca-bundle\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.042511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042444    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6txqt\" (UniqueName: \"kubernetes.io/projected/e9b403df-9c62-4d05-b8cc-2d2e3d94f521-kube-api-access-6txqt\") pod \"console-operator-696884fb9c-mxm76\" (UID: \"e9b403df-9c62-4d05-b8cc-2d2e3d94f521\") " pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:36.042511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-serving-cert\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.042884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042514    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-default-certificate\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:36.042884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042558    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mjv99\" (UID: \"d3828f83-14bf-4688-ba93-b65e204a4118\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"
Sep 16 18:16:36.042884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da984918-11f3-4d23-9381-bd3cc174771f-ca-trust-extracted\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.042884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042618    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd5fl\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-kube-api-access-vd5fl\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.042884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042658    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:36.042884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042683    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da984918-11f3-4d23-9381-bd3cc174771f-installation-pull-secrets\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.042884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.042724    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-stats-auth\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:36.143127 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06266e44-9985-4fdc-af4f-1476cabba0fd-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-frd2q\" (UID: \"06266e44-9985-4fdc-af4f-1476cabba0fd\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:16:36.143127 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-bound-sa-token\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.143376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143143    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrwgb\" (UniqueName: \"kubernetes.io/projected/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-kube-api-access-hrwgb\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.143376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143164    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9b403df-9c62-4d05-b8cc-2d2e3d94f521-serving-cert\") pod \"console-operator-696884fb9c-mxm76\" (UID: \"e9b403df-9c62-4d05-b8cc-2d2e3d94f521\") " pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:36.143376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143188    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d69kn\" (UniqueName: \"kubernetes.io/projected/06266e44-9985-4fdc-af4f-1476cabba0fd-kube-api-access-d69kn\") pod \"cluster-monitoring-operator-6cd954b897-frd2q\" (UID: \"06266e44-9985-4fdc-af4f-1476cabba0fd\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:16:36.143376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-tmp\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.143376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-service-ca-bundle\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.143376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143259    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6txqt\" (UniqueName: \"kubernetes.io/projected/e9b403df-9c62-4d05-b8cc-2d2e3d94f521-kube-api-access-6txqt\") pod \"console-operator-696884fb9c-mxm76\" (UID: \"e9b403df-9c62-4d05-b8cc-2d2e3d94f521\") " pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:36.143376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-serving-cert\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.143376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-default-certificate\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:36.143376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143349    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mjv99\" (UID: \"d3828f83-14bf-4688-ba93-b65e204a4118\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"
Sep 16 18:16:36.143376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143375    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da984918-11f3-4d23-9381-bd3cc174771f-ca-trust-extracted\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143400    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd5fl\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-kube-api-access-vd5fl\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143435    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6340b4-d303-4081-b1c6-c12e28cb1a31-config-volume\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143458    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b6340b4-d303-4081-b1c6-c12e28cb1a31-tmp-dir\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143495    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da984918-11f3-4d23-9381-bd3cc174771f-installation-pull-secrets\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-stats-auth\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da984918-11f3-4d23-9381-bd3cc174771f-trusted-ca\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da984918-11f3-4d23-9381-bd3cc174771f-registry-certificates\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143672    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143697    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfx9j\" (UniqueName: \"kubernetes.io/projected/743d0ab1-2506-4fba-951d-80eee5871235-kube-api-access-xfx9j\") pod \"ingress-canary-vnc87\" (UID: \"743d0ab1-2506-4fba-951d-80eee5871235\") " pod="openshift-ingress-canary/ingress-canary-vnc87"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143771    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzk8v\" (UniqueName: \"kubernetes.io/projected/92d1ca76-661a-47d9-bb84-e11ee89825e7-kube-api-access-tzk8v\") pod \"volume-data-source-validator-86695b7cc7-gcvrf\" (UID: \"92d1ca76-661a-47d9-bb84-e11ee89825e7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcvrf"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143782    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-tmp\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143802    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-trusted-ca-bundle\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143831    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qxg8\" (UniqueName: \"kubernetes.io/projected/67a74600-f466-4ae4-b4bf-67ab02745509-kube-api-access-4qxg8\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:36.143878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143863    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da984918-11f3-4d23-9381-bd3cc174771f-image-registry-private-configuration\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143888    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143912    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh248\" (UniqueName: \"kubernetes.io/projected/d3828f83-14bf-4688-ba93-b65e204a4118-kube-api-access-bh248\") pod \"cluster-samples-operator-86499c89c4-mjv99\" (UID: \"d3828f83-14bf-4688-ba93-b65e204a4118\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.143781    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143957    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-frd2q\" (UID: \"06266e44-9985-4fdc-af4f-1476cabba0fd\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.143983    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/743d0ab1-2506-4fba-951d-80eee5871235-cert\") pod \"ingress-canary-vnc87\" (UID: \"743d0ab1-2506-4fba-951d-80eee5871235\") " pod="openshift-ingress-canary/ingress-canary-vnc87"
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.144031    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-snapshots\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.144070    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls podName:d3828f83-14bf-4688-ba93-b65e204a4118 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:36.644047362 +0000 UTC m=+34.508951397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-mjv99" (UID: "d3828f83-14bf-4688-ba93-b65e204a4118") : secret "samples-operator-tls" not found
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.144100    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9b403df-9c62-4d05-b8cc-2d2e3d94f521-config\") pod \"console-operator-696884fb9c-mxm76\" (UID: \"e9b403df-9c62-4d05-b8cc-2d2e3d94f521\") " pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.144125    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da984918-11f3-4d23-9381-bd3cc174771f-ca-trust-extracted\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.144138    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9b403df-9c62-4d05-b8cc-2d2e3d94f521-trusted-ca\") pod \"console-operator-696884fb9c-mxm76\" (UID: \"e9b403df-9c62-4d05-b8cc-2d2e3d94f521\") " pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.144172    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnv5m\" (UniqueName: \"kubernetes.io/projected/1b6340b4-d303-4081-b1c6-c12e28cb1a31-kube-api-access-mnv5m\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.144361    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle podName:67a74600-f466-4ae4-b4bf-67ab02745509 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:36.64434262 +0000 UTC m=+34.509246665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle") pod "router-default-c65ff548d-bxgzt" (UID: "67a74600-f466-4ae4-b4bf-67ab02745509") : configmap references non-existent config key: service-ca.crt
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.144369    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06266e44-9985-4fdc-af4f-1476cabba0fd-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-frd2q\" (UID: \"06266e44-9985-4fdc-af4f-1476cabba0fd\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.144487    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.144503    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f6855565-qv2zc: secret "image-registry-tls" not found
Sep 16 18:16:36.144770 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.144619    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:16:36.147326 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.144790    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:16:36.147326 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.145305    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-service-ca-bundle\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.147326 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.145523    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-trusted-ca-bundle\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.147326 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.146288    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da984918-11f3-4d23-9381-bd3cc174771f-trusted-ca\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.147326 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.146321    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls podName:da984918-11f3-4d23-9381-bd3cc174771f nodeName:}" failed. No retries permitted until 2026-09-16 18:16:36.646286144 +0000 UTC m=+34.511190177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls") pod "image-registry-84f6855565-qv2zc" (UID: "da984918-11f3-4d23-9381-bd3cc174771f") : secret "image-registry-tls" not found
Sep 16 18:16:36.147326 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.146355    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls podName:06266e44-9985-4fdc-af4f-1476cabba0fd nodeName:}" failed. No retries permitted until 2026-09-16 18:16:36.646344018 +0000 UTC m=+34.511248056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-frd2q" (UID: "06266e44-9985-4fdc-af4f-1476cabba0fd") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:16:36.147326 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.146374    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs podName:67a74600-f466-4ae4-b4bf-67ab02745509 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:36.646365421 +0000 UTC m=+34.511269456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs") pod "router-default-c65ff548d-bxgzt" (UID: "67a74600-f466-4ae4-b4bf-67ab02745509") : secret "router-metrics-certs-default" not found
Sep 16 18:16:36.147326 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.146405    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e9b403df-9c62-4d05-b8cc-2d2e3d94f521-config\") pod \"console-operator-696884fb9c-mxm76\" (UID: \"e9b403df-9c62-4d05-b8cc-2d2e3d94f521\") " pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:36.147326 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.146678    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-snapshots\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.147326 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.146726    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e9b403df-9c62-4d05-b8cc-2d2e3d94f521-trusted-ca\") pod \"console-operator-696884fb9c-mxm76\" (UID: \"e9b403df-9c62-4d05-b8cc-2d2e3d94f521\") " pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:36.147326 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.146754    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da984918-11f3-4d23-9381-bd3cc174771f-registry-certificates\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.149301 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.149261    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da984918-11f3-4d23-9381-bd3cc174771f-image-registry-private-configuration\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.149301 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.149281    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e9b403df-9c62-4d05-b8cc-2d2e3d94f521-serving-cert\") pod \"console-operator-696884fb9c-mxm76\" (UID: \"e9b403df-9c62-4d05-b8cc-2d2e3d94f521\") " pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:36.149417 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.149395    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-default-certificate\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:36.149453 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.149420    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-stats-auth\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:36.149952 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.149926    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-serving-cert\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.150232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.150214    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da984918-11f3-4d23-9381-bd3cc174771f-installation-pull-secrets\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.156912 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.156887    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d69kn\" (UniqueName: \"kubernetes.io/projected/06266e44-9985-4fdc-af4f-1476cabba0fd-kube-api-access-d69kn\") pod \"cluster-monitoring-operator-6cd954b897-frd2q\" (UID: \"06266e44-9985-4fdc-af4f-1476cabba0fd\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:16:36.166514 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.166446    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzk8v\" (UniqueName: \"kubernetes.io/projected/92d1ca76-661a-47d9-bb84-e11ee89825e7-kube-api-access-tzk8v\") pod \"volume-data-source-validator-86695b7cc7-gcvrf\" (UID: \"92d1ca76-661a-47d9-bb84-e11ee89825e7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcvrf"
Sep 16 18:16:36.166677 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.166661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd5fl\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-kube-api-access-vd5fl\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.166843 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.166821    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh248\" (UniqueName: \"kubernetes.io/projected/d3828f83-14bf-4688-ba93-b65e204a4118-kube-api-access-bh248\") pod \"cluster-samples-operator-86499c89c4-mjv99\" (UID: \"d3828f83-14bf-4688-ba93-b65e204a4118\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"
Sep 16 18:16:36.167049 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.166997    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrwgb\" (UniqueName: \"kubernetes.io/projected/4642a843-fe4e-43a4-ab7e-64cc1c66bf10-kube-api-access-hrwgb\") pod \"insights-operator-5844b856db-wnqkf\" (UID: \"4642a843-fe4e-43a4-ab7e-64cc1c66bf10\") " pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.167049 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.166999    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qxg8\" (UniqueName: \"kubernetes.io/projected/67a74600-f466-4ae4-b4bf-67ab02745509-kube-api-access-4qxg8\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:36.167484 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.167462    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6txqt\" (UniqueName: \"kubernetes.io/projected/e9b403df-9c62-4d05-b8cc-2d2e3d94f521-kube-api-access-6txqt\") pod \"console-operator-696884fb9c-mxm76\" (UID: \"e9b403df-9c62-4d05-b8cc-2d2e3d94f521\") " pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:36.167911 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.167889    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-bound-sa-token\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.230830 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.230782    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:36.245013 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.244977    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/743d0ab1-2506-4fba-951d-80eee5871235-cert\") pod \"ingress-canary-vnc87\" (UID: \"743d0ab1-2506-4fba-951d-80eee5871235\") " pod="openshift-ingress-canary/ingress-canary-vnc87"
Sep 16 18:16:36.245492 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.245024    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnv5m\" (UniqueName: \"kubernetes.io/projected/1b6340b4-d303-4081-b1c6-c12e28cb1a31-kube-api-access-mnv5m\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:36.245492 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.245087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6340b4-d303-4081-b1c6-c12e28cb1a31-config-volume\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:36.245492 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.245105    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b6340b4-d303-4081-b1c6-c12e28cb1a31-tmp-dir\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:36.245492 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.245174    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:36.245492 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.245190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfx9j\" (UniqueName: \"kubernetes.io/projected/743d0ab1-2506-4fba-951d-80eee5871235-kube-api-access-xfx9j\") pod \"ingress-canary-vnc87\" (UID: \"743d0ab1-2506-4fba-951d-80eee5871235\") " pod="openshift-ingress-canary/ingress-canary-vnc87"
Sep 16 18:16:36.245492 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.245388    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:16:36.245492 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.245455    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls podName:1b6340b4-d303-4081-b1c6-c12e28cb1a31 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:36.745432747 +0000 UTC m=+34.610336775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls") pod "dns-default-bcg6p" (UID: "1b6340b4-d303-4081-b1c6-c12e28cb1a31") : secret "dns-default-metrics-tls" not found
Sep 16 18:16:36.246038 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.245625    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b6340b4-d303-4081-b1c6-c12e28cb1a31-tmp-dir\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:36.246038 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.245916    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b6340b4-d303-4081-b1c6-c12e28cb1a31-config-volume\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:36.248036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.248010    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/743d0ab1-2506-4fba-951d-80eee5871235-cert\") pod \"ingress-canary-vnc87\" (UID: \"743d0ab1-2506-4fba-951d-80eee5871235\") " pod="openshift-ingress-canary/ingress-canary-vnc87"
Sep 16 18:16:36.257844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.257812    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfx9j\" (UniqueName: \"kubernetes.io/projected/743d0ab1-2506-4fba-951d-80eee5871235-kube-api-access-xfx9j\") pod \"ingress-canary-vnc87\" (UID: \"743d0ab1-2506-4fba-951d-80eee5871235\") " pod="openshift-ingress-canary/ingress-canary-vnc87"
Sep 16 18:16:36.257844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.257816    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnv5m\" (UniqueName: \"kubernetes.io/projected/1b6340b4-d303-4081-b1c6-c12e28cb1a31-kube-api-access-mnv5m\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:36.311228 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.311192    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcvrf"
Sep 16 18:16:36.320326 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.320296    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-wnqkf"
Sep 16 18:16:36.334391 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.334351    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:36.342124 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.342096    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:36.356446 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.356415    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vnc87"
Sep 16 18:16:36.543751 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.543656    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:16:36.547020 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.546980    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:16:36.547178 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.547023    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xdwg2\""
Sep 16 18:16:36.648524 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.648488    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mjv99\" (UID: \"d3828f83-14bf-4688-ba93-b65e204a4118\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"
Sep 16 18:16:36.648723 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.648598    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:36.648723 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.648654    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:16:36.648723 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.648677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:36.648723 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.648722    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls podName:d3828f83-14bf-4688-ba93-b65e204a4118 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:37.648701126 +0000 UTC m=+35.513605168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-mjv99" (UID: "d3828f83-14bf-4688-ba93-b65e204a4118") : secret "samples-operator-tls" not found
Sep 16 18:16:36.648917 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.648758    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:16:36.648917 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.648763    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:36.648917 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.648819    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:16:36.648917 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.648832    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f6855565-qv2zc: secret "image-registry-tls" not found
Sep 16 18:16:36.648917 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.648819    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs podName:67a74600-f466-4ae4-b4bf-67ab02745509 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:37.64880215 +0000 UTC m=+35.513706193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs") pod "router-default-c65ff548d-bxgzt" (UID: "67a74600-f466-4ae4-b4bf-67ab02745509") : secret "router-metrics-certs-default" not found
Sep 16 18:16:36.652379 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.649300    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle podName:67a74600-f466-4ae4-b4bf-67ab02745509 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:37.648871957 +0000 UTC m=+35.513775990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle") pod "router-default-c65ff548d-bxgzt" (UID: "67a74600-f466-4ae4-b4bf-67ab02745509") : configmap references non-existent config key: service-ca.crt
Sep 16 18:16:36.652379 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.649351    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls podName:da984918-11f3-4d23-9381-bd3cc174771f nodeName:}" failed. No retries permitted until 2026-09-16 18:16:37.649329855 +0000 UTC m=+35.514233887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls") pod "image-registry-84f6855565-qv2zc" (UID: "da984918-11f3-4d23-9381-bd3cc174771f") : secret "image-registry-tls" not found
Sep 16 18:16:36.652379 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.649419    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-frd2q\" (UID: \"06266e44-9985-4fdc-af4f-1476cabba0fd\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:16:36.652379 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.649614    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:16:36.652379 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.649674    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls podName:06266e44-9985-4fdc-af4f-1476cabba0fd nodeName:}" failed. No retries permitted until 2026-09-16 18:16:37.64965817 +0000 UTC m=+35.514562212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-frd2q" (UID: "06266e44-9985-4fdc-af4f-1476cabba0fd") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:16:36.750368 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:36.750325    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:36.750594 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.750509    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:16:36.750678 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:36.750605    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls podName:1b6340b4-d303-4081-b1c6-c12e28cb1a31 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:37.750582974 +0000 UTC m=+35.615487016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls") pod "dns-default-bcg6p" (UID: "1b6340b4-d303-4081-b1c6-c12e28cb1a31") : secret "dns-default-metrics-tls" not found
Sep 16 18:16:37.543526 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.543483    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:16:37.544040 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.543483    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:16:37.552111 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.551412    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gs7dj\""
Sep 16 18:16:37.552111 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.551465    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:16:37.552111 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.551507    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:16:37.552111 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.551528    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:16:37.620412 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.620383    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4"]
Sep 16 18:16:37.658559 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.658502    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4"]
Sep 16 18:16:37.658758 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.658644    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4"
Sep 16 18:16:37.658830 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.658775    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:37.658830 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.658813    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:37.658921 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.658857    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-frd2q\" (UID: \"06266e44-9985-4fdc-af4f-1476cabba0fd\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:16:37.658969 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.658920    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mjv99\" (UID: \"d3828f83-14bf-4688-ba93-b65e204a4118\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"
Sep 16 18:16:37.658969 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.658960    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:37.659066 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:37.659015    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:16:37.659066 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:37.659045    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:16:37.659155 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:37.659078    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls podName:06266e44-9985-4fdc-af4f-1476cabba0fd nodeName:}" failed. No retries permitted until 2026-09-16 18:16:39.659057923 +0000 UTC m=+37.523961962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-frd2q" (UID: "06266e44-9985-4fdc-af4f-1476cabba0fd") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:16:37.659155 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:37.659019    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:16:37.659155 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:37.659096    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f6855565-qv2zc: secret "image-registry-tls" not found
Sep 16 18:16:37.659155 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:37.659098    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs podName:67a74600-f466-4ae4-b4bf-67ab02745509 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:39.659088705 +0000 UTC m=+37.523992731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs") pod "router-default-c65ff548d-bxgzt" (UID: "67a74600-f466-4ae4-b4bf-67ab02745509") : secret "router-metrics-certs-default" not found
Sep 16 18:16:37.659155 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:37.659115    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle podName:67a74600-f466-4ae4-b4bf-67ab02745509 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:39.659106722 +0000 UTC m=+37.524010751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle") pod "router-default-c65ff548d-bxgzt" (UID: "67a74600-f466-4ae4-b4bf-67ab02745509") : configmap references non-existent config key: service-ca.crt
Sep 16 18:16:37.659367 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:37.659160    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls podName:da984918-11f3-4d23-9381-bd3cc174771f nodeName:}" failed. No retries permitted until 2026-09-16 18:16:39.65915131 +0000 UTC m=+37.524055338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls") pod "image-registry-84f6855565-qv2zc" (UID: "da984918-11f3-4d23-9381-bd3cc174771f") : secret "image-registry-tls" not found
Sep 16 18:16:37.659367 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:37.659162    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:16:37.659367 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:37.659239    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls podName:d3828f83-14bf-4688-ba93-b65e204a4118 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:39.659199369 +0000 UTC m=+37.524103399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-mjv99" (UID: "d3828f83-14bf-4688-ba93-b65e204a4118") : secret "samples-operator-tls" not found
Sep 16 18:16:37.662697 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.662669    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 18:16:37.662697 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.662679    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:16:37.662697 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.662669    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-kvf4g\""
Sep 16 18:16:37.662942 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.662669    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 18:16:37.663094 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.663077    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 18:16:37.760118 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.759853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a69f057-7c16-42af-8e69-9be2c06c02c7-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wrll4\" (UID: \"4a69f057-7c16-42af-8e69-9be2c06c02c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4"
Sep 16 18:16:37.760366 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.760141    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vd49\" (UniqueName: \"kubernetes.io/projected/4a69f057-7c16-42af-8e69-9be2c06c02c7-kube-api-access-2vd49\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wrll4\" (UID: \"4a69f057-7c16-42af-8e69-9be2c06c02c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4"
Sep 16 18:16:37.760366 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.760266    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:37.760495 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.760387    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a69f057-7c16-42af-8e69-9be2c06c02c7-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wrll4\" (UID: \"4a69f057-7c16-42af-8e69-9be2c06c02c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4"
Sep 16 18:16:37.760495 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:37.760430    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:16:37.760601 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:37.760527    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls podName:1b6340b4-d303-4081-b1c6-c12e28cb1a31 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:39.760503046 +0000 UTC m=+37.625407087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls") pod "dns-default-bcg6p" (UID: "1b6340b4-d303-4081-b1c6-c12e28cb1a31") : secret "dns-default-metrics-tls" not found
Sep 16 18:16:37.861480 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.861436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a69f057-7c16-42af-8e69-9be2c06c02c7-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wrll4\" (UID: \"4a69f057-7c16-42af-8e69-9be2c06c02c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4"
Sep 16 18:16:37.861683 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.861508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a69f057-7c16-42af-8e69-9be2c06c02c7-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wrll4\" (UID: \"4a69f057-7c16-42af-8e69-9be2c06c02c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4"
Sep 16 18:16:37.861683 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.861529    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vd49\" (UniqueName: \"kubernetes.io/projected/4a69f057-7c16-42af-8e69-9be2c06c02c7-kube-api-access-2vd49\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wrll4\" (UID: \"4a69f057-7c16-42af-8e69-9be2c06c02c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4"
Sep 16 18:16:37.862083 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.862063    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4a69f057-7c16-42af-8e69-9be2c06c02c7-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wrll4\" (UID: \"4a69f057-7c16-42af-8e69-9be2c06c02c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4"
Sep 16 18:16:37.864640 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.864609    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a69f057-7c16-42af-8e69-9be2c06c02c7-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wrll4\" (UID: \"4a69f057-7c16-42af-8e69-9be2c06c02c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4"
Sep 16 18:16:37.870864 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.870836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vd49\" (UniqueName: \"kubernetes.io/projected/4a69f057-7c16-42af-8e69-9be2c06c02c7-kube-api-access-2vd49\") pod \"kube-storage-version-migrator-operator-6c8f84f897-wrll4\" (UID: \"4a69f057-7c16-42af-8e69-9be2c06c02c7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4"
Sep 16 18:16:37.967503 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:37.967470    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4"
Sep 16 18:16:38.031233 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.031187    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcvrf"]
Sep 16 18:16:38.042027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.035729    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-wnqkf"]
Sep 16 18:16:38.042027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.037415    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vnc87"]
Sep 16 18:16:38.042027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.038600    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-mxm76"]
Sep 16 18:16:38.042027 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:38.040160    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d1ca76_661a_47d9_bb84_e11ee89825e7.slice/crio-1f7096711588460332c35a4f63ac05eecff4819567406d12e49018e6af8d7e34 WatchSource:0}: Error finding container 1f7096711588460332c35a4f63ac05eecff4819567406d12e49018e6af8d7e34: Status 404 returned error can't find the container with id 1f7096711588460332c35a4f63ac05eecff4819567406d12e49018e6af8d7e34
Sep 16 18:16:38.045913 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:38.045705    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743d0ab1_2506_4fba_951d_80eee5871235.slice/crio-65a7c8cb008022697e1edf50965e14573dbf4e61a48ded06f243163516d39ba6 WatchSource:0}: Error finding container 65a7c8cb008022697e1edf50965e14573dbf4e61a48ded06f243163516d39ba6: Status 404 returned error can't find the container with id 65a7c8cb008022697e1edf50965e14573dbf4e61a48ded06f243163516d39ba6
Sep 16 18:16:38.046867 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:38.046365    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b403df_9c62_4d05_b8cc_2d2e3d94f521.slice/crio-5a9649f71bff65daf50c9c23f1128451e353f367de8b879aca02da0d7939eb24 WatchSource:0}: Error finding container 5a9649f71bff65daf50c9c23f1128451e353f367de8b879aca02da0d7939eb24: Status 404 returned error can't find the container with id 5a9649f71bff65daf50c9c23f1128451e353f367de8b879aca02da0d7939eb24
Sep 16 18:16:38.049733 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:38.048468    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4642a843_fe4e_43a4_ab7e_64cc1c66bf10.slice/crio-e48e59282698a63c4a4c76dd9babd761d3eadb46dc147f9a10d15ff410bcb5ed WatchSource:0}: Error finding container e48e59282698a63c4a4c76dd9babd761d3eadb46dc147f9a10d15ff410bcb5ed: Status 404 returned error can't find the container with id e48e59282698a63c4a4c76dd9babd761d3eadb46dc147f9a10d15ff410bcb5ed
Sep 16 18:16:38.264341 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.264309    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4"]
Sep 16 18:16:38.267245 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:16:38.267210    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a69f057_7c16_42af_8e69_9be2c06c02c7.slice/crio-45bd35cf905d560cedb76ce039fc6a7bb6240fd22007520aeb2c999168c9da82 WatchSource:0}: Error finding container 45bd35cf905d560cedb76ce039fc6a7bb6240fd22007520aeb2c999168c9da82: Status 404 returned error can't find the container with id 45bd35cf905d560cedb76ce039fc6a7bb6240fd22007520aeb2c999168c9da82
Sep 16 18:16:38.268703 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.268671    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:38.344472 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.344423    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:38.443814 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.443770    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:38.774255 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.774040    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mxm76" event={"ID":"e9b403df-9c62-4d05-b8cc-2d2e3d94f521","Type":"ContainerStarted","Data":"5a9649f71bff65daf50c9c23f1128451e353f367de8b879aca02da0d7939eb24"}
Sep 16 18:16:38.775521 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.775492    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcvrf" event={"ID":"92d1ca76-661a-47d9-bb84-e11ee89825e7","Type":"ContainerStarted","Data":"1f7096711588460332c35a4f63ac05eecff4819567406d12e49018e6af8d7e34"}
Sep 16 18:16:38.776988 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.776933    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wnqkf" event={"ID":"4642a843-fe4e-43a4-ab7e-64cc1c66bf10","Type":"ContainerStarted","Data":"e48e59282698a63c4a4c76dd9babd761d3eadb46dc147f9a10d15ff410bcb5ed"}
Sep 16 18:16:38.778602 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.778570    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4" event={"ID":"4a69f057-7c16-42af-8e69-9be2c06c02c7","Type":"ContainerStarted","Data":"45bd35cf905d560cedb76ce039fc6a7bb6240fd22007520aeb2c999168c9da82"}
Sep 16 18:16:38.782382 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.782236    2562 generic.go:356] "Generic (PLEG): container finished" podID="fb9d634a-bee5-4019-8a89-32065530f04c" containerID="26ec9d1346f7fac703b2686aa378757a15dc00ec01d38ab67a1b925eb55d7c66" exitCode=0
Sep 16 18:16:38.782382 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.782328    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wqd" event={"ID":"fb9d634a-bee5-4019-8a89-32065530f04c","Type":"ContainerDied","Data":"26ec9d1346f7fac703b2686aa378757a15dc00ec01d38ab67a1b925eb55d7c66"}
Sep 16 18:16:38.784526 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.784469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vnc87" event={"ID":"743d0ab1-2506-4fba-951d-80eee5871235","Type":"ContainerStarted","Data":"65a7c8cb008022697e1edf50965e14573dbf4e61a48ded06f243163516d39ba6"}
Sep 16 18:16:38.922902 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.922648    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-84mpz"]
Sep 16 18:16:38.943731 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.943243    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-84mpz"]
Sep 16 18:16:38.943731 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.943378    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-84mpz"
Sep 16 18:16:38.948481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:38.948454    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-48jp5\""
Sep 16 18:16:39.072464 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.072276    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nq99\" (UniqueName: \"kubernetes.io/projected/575615e6-3927-48ae-b9f1-83a34defb73e-kube-api-access-8nq99\") pod \"network-check-source-6f8b768455-84mpz\" (UID: \"575615e6-3927-48ae-b9f1-83a34defb73e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-84mpz"
Sep 16 18:16:39.174022 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.173577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nq99\" (UniqueName: \"kubernetes.io/projected/575615e6-3927-48ae-b9f1-83a34defb73e-kube-api-access-8nq99\") pod \"network-check-source-6f8b768455-84mpz\" (UID: \"575615e6-3927-48ae-b9f1-83a34defb73e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-84mpz"
Sep 16 18:16:39.188636 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.188567    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nq99\" (UniqueName: \"kubernetes.io/projected/575615e6-3927-48ae-b9f1-83a34defb73e-kube-api-access-8nq99\") pod \"network-check-source-6f8b768455-84mpz\" (UID: \"575615e6-3927-48ae-b9f1-83a34defb73e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-84mpz"
Sep 16 18:16:39.258758 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.258453    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-84mpz"
Sep 16 18:16:39.308585 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.308526    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:39.457790 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.457754    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:39.583119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.583017    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-84mpz"]
Sep 16 18:16:39.678603 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.678554    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:39.678851 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.678629    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:39.678851 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.678680    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-frd2q\" (UID: \"06266e44-9985-4fdc-af4f-1476cabba0fd\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:16:39.678851 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.678754    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mjv99\" (UID: \"d3828f83-14bf-4688-ba93-b65e204a4118\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"
Sep 16 18:16:39.678851 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.678806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:39.679063 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:39.679001    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:16:39.680256 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:39.679254    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:16:39.680256 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:39.679273    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f6855565-qv2zc: secret "image-registry-tls" not found
Sep 16 18:16:39.680256 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:39.679342    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls podName:da984918-11f3-4d23-9381-bd3cc174771f nodeName:}" failed. No retries permitted until 2026-09-16 18:16:43.679320472 +0000 UTC m=+41.544224512 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls") pod "image-registry-84f6855565-qv2zc" (UID: "da984918-11f3-4d23-9381-bd3cc174771f") : secret "image-registry-tls" not found
Sep 16 18:16:39.680256 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:39.679431    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle podName:67a74600-f466-4ae4-b4bf-67ab02745509 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:43.679419262 +0000 UTC m=+41.544323303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle") pod "router-default-c65ff548d-bxgzt" (UID: "67a74600-f466-4ae4-b4bf-67ab02745509") : configmap references non-existent config key: service-ca.crt
Sep 16 18:16:39.680256 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:39.679711    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:16:39.680256 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:39.679761    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls podName:06266e44-9985-4fdc-af4f-1476cabba0fd nodeName:}" failed. No retries permitted until 2026-09-16 18:16:43.679745525 +0000 UTC m=+41.544649558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-frd2q" (UID: "06266e44-9985-4fdc-af4f-1476cabba0fd") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:16:39.680256 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:39.679830    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:16:39.680256 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:39.679863    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls podName:d3828f83-14bf-4688-ba93-b65e204a4118 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:43.679851828 +0000 UTC m=+41.544755867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-mjv99" (UID: "d3828f83-14bf-4688-ba93-b65e204a4118") : secret "samples-operator-tls" not found
Sep 16 18:16:39.680838 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:39.680801    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs podName:67a74600-f466-4ae4-b4bf-67ab02745509 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:43.680775561 +0000 UTC m=+41.545679601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs") pod "router-default-c65ff548d-bxgzt" (UID: "67a74600-f466-4ae4-b4bf-67ab02745509") : secret "router-metrics-certs-default" not found
Sep 16 18:16:39.780124 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.779886    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:39.780124 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:39.780105    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:16:39.780805 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:39.780177    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls podName:1b6340b4-d303-4081-b1c6-c12e28cb1a31 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:43.780156711 +0000 UTC m=+41.645060737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls") pod "dns-default-bcg6p" (UID: "1b6340b4-d303-4081-b1c6-c12e28cb1a31") : secret "dns-default-metrics-tls" not found
Sep 16 18:16:39.792212 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.792129    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-84mpz" event={"ID":"575615e6-3927-48ae-b9f1-83a34defb73e","Type":"ContainerStarted","Data":"10519b322cf754309a2be665d458c4c554341cf94eafcdc15a5206c28aa40d20"}
Sep 16 18:16:39.798288 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.798251    2562 generic.go:356] "Generic (PLEG): container finished" podID="fb9d634a-bee5-4019-8a89-32065530f04c" containerID="9ba4d686821868cef491e68065208d7c9c4ce9a55e2db9d69b9dfd14282bedfc" exitCode=0
Sep 16 18:16:39.798476 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:39.798311    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wqd" event={"ID":"fb9d634a-bee5-4019-8a89-32065530f04c","Type":"ContainerDied","Data":"9ba4d686821868cef491e68065208d7c9c4ce9a55e2db9d69b9dfd14282bedfc"}
Sep 16 18:16:40.452008 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:40.451954    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:40.583951 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:40.583910    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:40.806821 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:40.806738    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wqd" event={"ID":"fb9d634a-bee5-4019-8a89-32065530f04c","Type":"ContainerStarted","Data":"9891c55648f6588a1940bf2e6c8314ea69719f6fafe64bef60002b72e9c57781"}
Sep 16 18:16:40.850806 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:40.850641    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-52wqd" podStartSLOduration=8.21529961 podStartE2EDuration="38.850619994s" totalImagesPullingTime="30.635320384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:40.848995107 +0000 UTC m=+38.713899177" watchObservedRunningTime="2026-09-16 18:16:40.850619994 +0000 UTC m=+38.715524050"
Sep 16 18:16:40.988228 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:40.988196    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:41.072151 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:41.072053    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:41.469135 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:41.469098    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:41.565614 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:41.565574    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:43.260787 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:43.260746    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:43.357740 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:43.357709    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:43.458490 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:43.458453    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:43.718658 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:43.718613    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:43.718821 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:43.718702    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:43.718821 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:43.718731    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:43.718821 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:43.718756    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:16:43.718821 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:43.718766    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-frd2q\" (UID: \"06266e44-9985-4fdc-af4f-1476cabba0fd\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:16:43.718821 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:43.718815    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs podName:67a74600-f466-4ae4-b4bf-67ab02745509 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:51.718799691 +0000 UTC m=+49.583703720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs") pod "router-default-c65ff548d-bxgzt" (UID: "67a74600-f466-4ae4-b4bf-67ab02745509") : secret "router-metrics-certs-default" not found
Sep 16 18:16:43.719035 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:43.718853    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mjv99\" (UID: \"d3828f83-14bf-4688-ba93-b65e204a4118\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"
Sep 16 18:16:43.719035 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:43.718864    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:16:43.719035 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:43.718870    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:16:43.719035 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:43.718886    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle podName:67a74600-f466-4ae4-b4bf-67ab02745509 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:51.718869723 +0000 UTC m=+49.583773783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle") pod "router-default-c65ff548d-bxgzt" (UID: "67a74600-f466-4ae4-b4bf-67ab02745509") : configmap references non-existent config key: service-ca.crt
Sep 16 18:16:43.719035 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:43.718887    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f6855565-qv2zc: secret "image-registry-tls" not found
Sep 16 18:16:43.719035 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:43.718909    2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:16:43.719035 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:43.718913    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls podName:06266e44-9985-4fdc-af4f-1476cabba0fd nodeName:}" failed. No retries permitted until 2026-09-16 18:16:51.718904638 +0000 UTC m=+49.583808673 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-frd2q" (UID: "06266e44-9985-4fdc-af4f-1476cabba0fd") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:16:43.719035 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:43.718945    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls podName:d3828f83-14bf-4688-ba93-b65e204a4118 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:51.718932767 +0000 UTC m=+49.583836807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-mjv99" (UID: "d3828f83-14bf-4688-ba93-b65e204a4118") : secret "samples-operator-tls" not found
Sep 16 18:16:43.719035 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:43.718970    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls podName:da984918-11f3-4d23-9381-bd3cc174771f nodeName:}" failed. No retries permitted until 2026-09-16 18:16:51.718954219 +0000 UTC m=+49.583858266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls") pod "image-registry-84f6855565-qv2zc" (UID: "da984918-11f3-4d23-9381-bd3cc174771f") : secret "image-registry-tls" not found
Sep 16 18:16:43.819967 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:43.819936    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:43.820133 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:43.820069    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:16:43.820192 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:43.820149    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls podName:1b6340b4-d303-4081-b1c6-c12e28cb1a31 nodeName:}" failed. No retries permitted until 2026-09-16 18:16:51.820128016 +0000 UTC m=+49.685032064 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls") pod "dns-default-bcg6p" (UID: "1b6340b4-d303-4081-b1c6-c12e28cb1a31") : secret "dns-default-metrics-tls" not found
Sep 16 18:16:44.552688 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:44.552634    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:44.626755 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:44.626718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:45.368714 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.368624    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:45.445002 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.444963    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:45.747683 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.747641    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:45.822722 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.822661    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:45.824850 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.824818    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/0.log"
Sep 16 18:16:45.825051 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.824869    2562 generic.go:356] "Generic (PLEG): container finished" podID="e9b403df-9c62-4d05-b8cc-2d2e3d94f521" containerID="fbe946fb06cd06dc4fb2d2e47507d977acabee2f029bea7add37e03b03166ab2" exitCode=255
Sep 16 18:16:45.825051 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.824914    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mxm76" event={"ID":"e9b403df-9c62-4d05-b8cc-2d2e3d94f521","Type":"ContainerDied","Data":"fbe946fb06cd06dc4fb2d2e47507d977acabee2f029bea7add37e03b03166ab2"}
Sep 16 18:16:45.825263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.825242    2562 scope.go:117] "RemoveContainer" containerID="fbe946fb06cd06dc4fb2d2e47507d977acabee2f029bea7add37e03b03166ab2"
Sep 16 18:16:45.826792 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.826756    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcvrf" event={"ID":"92d1ca76-661a-47d9-bb84-e11ee89825e7","Type":"ContainerStarted","Data":"9108297ba7400cc0e07987eeba7f0dafc7351ca0dc5dab488b8ed2b8c83b6507"}
Sep 16 18:16:45.828661 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.828635    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-84mpz" event={"ID":"575615e6-3927-48ae-b9f1-83a34defb73e","Type":"ContainerStarted","Data":"1ed60258bb739e0da89f904c452d1f25728e3df0f26d7f6bb7d9d5a3754e1064"}
Sep 16 18:16:45.830514 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.830328    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wnqkf" event={"ID":"4642a843-fe4e-43a4-ab7e-64cc1c66bf10","Type":"ContainerStarted","Data":"94ef66c98026b501afe377ad7283e64354fa1249df8d1cf040920a6d560201c6"}
Sep 16 18:16:45.831951 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.831925    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4" event={"ID":"4a69f057-7c16-42af-8e69-9be2c06c02c7","Type":"ContainerStarted","Data":"2a22163433e1e02bf20a4a73d3f16f81ef014465aa312775d25e4023275c847c"}
Sep 16 18:16:45.833658 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.833631    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vnc87" event={"ID":"743d0ab1-2506-4fba-951d-80eee5871235","Type":"ContainerStarted","Data":"1530d919121a380b2b8e9eb3872271fbf841a7996db4e8d62ae80fefd2663663"}
Sep 16 18:16:45.904372 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.904314    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gcvrf" podStartSLOduration=7.137886053 podStartE2EDuration="13.904295166s" totalImagesPullingTime="6.766409113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:45.87671165 +0000 UTC m=+43.741615700" watchObservedRunningTime="2026-09-16 18:16:45.904295166 +0000 UTC m=+43.769199214"
Sep 16 18:16:45.904696 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.904665    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vnc87" podStartSLOduration=4.142040352 podStartE2EDuration="10.904653602s" totalImagesPullingTime="6.76261325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:45.903938603 +0000 UTC m=+43.768842652" watchObservedRunningTime="2026-09-16 18:16:45.904653602 +0000 UTC m=+43.769557649"
Sep 16 18:16:45.942665 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:45.942409    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-wnqkf" podStartSLOduration=7.185109168 podStartE2EDuration="13.942388848s" totalImagesPullingTime="6.75727968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:45.9411366 +0000 UTC m=+43.806040649" watchObservedRunningTime="2026-09-16 18:16:45.942388848 +0000 UTC m=+43.807292901"
Sep 16 18:16:46.003013 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.002916    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4" podStartSLOduration=2.45985858 podStartE2EDuration="9.002894057s" totalImagesPullingTime="6.543035477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:46.001169774 +0000 UTC m=+43.866073823" watchObservedRunningTime="2026-09-16 18:16:46.002894057 +0000 UTC m=+43.867798106"
Sep 16 18:16:46.027870 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.027780    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-84mpz" podStartSLOduration=2.419803234 podStartE2EDuration="8.027755222s" totalImagesPullingTime="5.607951988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:46.026583043 +0000 UTC m=+43.891487091" watchObservedRunningTime="2026-09-16 18:16:46.027755222 +0000 UTC m=+43.892659271"
Sep 16 18:16:46.036727 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.036695    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:46.136698 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.136653    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:46.237687 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.237649    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:46.273238 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.273154    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p4k2k"]
Sep 16 18:16:46.297681 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.297644    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p4k2k"]
Sep 16 18:16:46.297860 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.297780    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:46.300715 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.300687    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-glhq5\""
Sep 16 18:16:46.300715 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.300694    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:16:46.300956 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.300943    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:16:46.342630 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.342600    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:46.342802 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.342655    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:46.422170 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.422133    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:46.446282 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.446251    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da9b816c-32e3-4d22-8940-806ebf564ffe-data-volume\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:46.446466 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.446393    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da9b816c-32e3-4d22-8940-806ebf564ffe-crio-socket\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:46.446466 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.446441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99cft\" (UniqueName: \"kubernetes.io/projected/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-api-access-99cft\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:46.446569 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.446470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:46.446569 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.446531    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:46.521174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.521137    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:46.547121 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.547004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:46.547121 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:46.547098    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:16:46.547304 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.547145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:46.547304 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:46.547157    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls podName:da9b816c-32e3-4d22-8940-806ebf564ffe nodeName:}" failed. No retries permitted until 2026-09-16 18:16:47.047140902 +0000 UTC m=+44.912044941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p4k2k" (UID: "da9b816c-32e3-4d22-8940-806ebf564ffe") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:16:46.547304 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.547199    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da9b816c-32e3-4d22-8940-806ebf564ffe-data-volume\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:46.547461 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.547311    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da9b816c-32e3-4d22-8940-806ebf564ffe-crio-socket\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:46.547461 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:46.547203    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:16:46.547461 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.547354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99cft\" (UniqueName: \"kubernetes.io/projected/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-api-access-99cft\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:46.547461 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:46.547365    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-rbac-proxy-cm podName:da9b816c-32e3-4d22-8940-806ebf564ffe nodeName:}" failed. No retries permitted until 2026-09-16 18:16:47.047351589 +0000 UTC m=+44.912255633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p4k2k" (UID: "da9b816c-32e3-4d22-8940-806ebf564ffe") : configmap "kube-rbac-proxy" not found
Sep 16 18:16:46.547696 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.547589    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da9b816c-32e3-4d22-8940-806ebf564ffe-crio-socket\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:46.547696 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.547606    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da9b816c-32e3-4d22-8940-806ebf564ffe-data-volume\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:46.564267 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.564231    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99cft\" (UniqueName: \"kubernetes.io/projected/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-api-access-99cft\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:46.621878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.621840    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:46.841017 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.840937    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:16:46.841436 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.841369    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/0.log"
Sep 16 18:16:46.841436 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.841403    2562 generic.go:356] "Generic (PLEG): container finished" podID="e9b403df-9c62-4d05-b8cc-2d2e3d94f521" containerID="124b932be39c7160afa11dd128947196ef5e03833c2e34febf97ce7790749b68" exitCode=255
Sep 16 18:16:46.841567 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.841490    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mxm76" event={"ID":"e9b403df-9c62-4d05-b8cc-2d2e3d94f521","Type":"ContainerDied","Data":"124b932be39c7160afa11dd128947196ef5e03833c2e34febf97ce7790749b68"}
Sep 16 18:16:46.841567 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.841565    2562 scope.go:117] "RemoveContainer" containerID="fbe946fb06cd06dc4fb2d2e47507d977acabee2f029bea7add37e03b03166ab2"
Sep 16 18:16:46.843296 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:46.842157    2562 scope.go:117] "RemoveContainer" containerID="124b932be39c7160afa11dd128947196ef5e03833c2e34febf97ce7790749b68"
Sep 16 18:16:46.843296 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:46.842502    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-mxm76_openshift-console-operator(e9b403df-9c62-4d05-b8cc-2d2e3d94f521)\"" pod="openshift-console-operator/console-operator-696884fb9c-mxm76" podUID="e9b403df-9c62-4d05-b8cc-2d2e3d94f521"
Sep 16 18:16:47.052864 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:47.052820    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:47.053050 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:47.052886    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:47.053050 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:47.052972    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:16:47.053050 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:47.053008    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:16:47.053050 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:47.053037    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls podName:da9b816c-32e3-4d22-8940-806ebf564ffe nodeName:}" failed. No retries permitted until 2026-09-16 18:16:48.053021626 +0000 UTC m=+45.917925653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p4k2k" (UID: "da9b816c-32e3-4d22-8940-806ebf564ffe") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:16:47.053237 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:47.053059    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-rbac-proxy-cm podName:da9b816c-32e3-4d22-8940-806ebf564ffe nodeName:}" failed. No retries permitted until 2026-09-16 18:16:48.05304742 +0000 UTC m=+45.917951459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p4k2k" (UID: "da9b816c-32e3-4d22-8940-806ebf564ffe") : configmap "kube-rbac-proxy" not found
Sep 16 18:16:47.846037 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:47.846008    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:16:47.846428 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:47.846336    2562 scope.go:117] "RemoveContainer" containerID="124b932be39c7160afa11dd128947196ef5e03833c2e34febf97ce7790749b68"
Sep 16 18:16:47.846503 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:47.846486    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-mxm76_openshift-console-operator(e9b403df-9c62-4d05-b8cc-2d2e3d94f521)\"" pod="openshift-console-operator/console-operator-696884fb9c-mxm76" podUID="e9b403df-9c62-4d05-b8cc-2d2e3d94f521"
Sep 16 18:16:47.928691 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:47.928587    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x5xdc_d5d1838b-753e-4591-b50a-473909c62656/dns-node-resolver/0.log"
Sep 16 18:16:48.062399 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:48.062359    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:48.062584 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:48.062419    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:48.062584 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:48.062534    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:16:48.062584 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:48.062578    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:16:48.062709 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:48.062634    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls podName:da9b816c-32e3-4d22-8940-806ebf564ffe nodeName:}" failed. No retries permitted until 2026-09-16 18:16:50.062613004 +0000 UTC m=+47.927517044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p4k2k" (UID: "da9b816c-32e3-4d22-8940-806ebf564ffe") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:16:48.062709 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:48.062652    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-rbac-proxy-cm podName:da9b816c-32e3-4d22-8940-806ebf564ffe nodeName:}" failed. No retries permitted until 2026-09-16 18:16:50.062643789 +0000 UTC m=+47.927547818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p4k2k" (UID: "da9b816c-32e3-4d22-8940-806ebf564ffe") : configmap "kube-rbac-proxy" not found
Sep 16 18:16:48.145324 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:48.145287    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:48.528807 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:48.528781    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-65kl9_e04d0991-aae5-4412-850d-6e4685c0124a/node-ca/0.log"
Sep 16 18:16:49.327016 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:49.326977    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:49.528592 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:49.528567    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vnc87_743d0ab1-2506-4fba-951d-80eee5871235/serve-healthcheck-canary/0.log"
Sep 16 18:16:50.080665 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:50.080627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:50.080860 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:50.080696    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:50.080860 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:50.080802    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:16:50.080860 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:50.080816    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:16:50.080972 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:50.080884    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-rbac-proxy-cm podName:da9b816c-32e3-4d22-8940-806ebf564ffe nodeName:}" failed. No retries permitted until 2026-09-16 18:16:54.080866813 +0000 UTC m=+51.945770843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p4k2k" (UID: "da9b816c-32e3-4d22-8940-806ebf564ffe") : configmap "kube-rbac-proxy" not found
Sep 16 18:16:50.080972 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:50.080898    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls podName:da9b816c-32e3-4d22-8940-806ebf564ffe nodeName:}" failed. No retries permitted until 2026-09-16 18:16:54.080892489 +0000 UTC m=+51.945796515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p4k2k" (UID: "da9b816c-32e3-4d22-8940-806ebf564ffe") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:16:51.492075 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:51.492041    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:51.633159 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:51.633119    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:51.725750 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:51.725714    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:51.794964 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:51.794866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:51.794964 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:51.794912    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:16:51.794964 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:51.794940    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-frd2q\" (UID: \"06266e44-9985-4fdc-af4f-1476cabba0fd\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:16:51.795221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:51.794987    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mjv99\" (UID: \"d3828f83-14bf-4688-ba93-b65e204a4118\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"
Sep 16 18:16:51.795221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:51.795020    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:16:51.795221 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:51.795043    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle podName:67a74600-f466-4ae4-b4bf-67ab02745509 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:07.795023871 +0000 UTC m=+65.659927913 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle") pod "router-default-c65ff548d-bxgzt" (UID: "67a74600-f466-4ae4-b4bf-67ab02745509") : configmap references non-existent config key: service-ca.crt
Sep 16 18:16:51.795445 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:51.795052    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:16:51.795499 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:51.795459    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f6855565-qv2zc: secret "image-registry-tls" not found
Sep 16 18:16:51.795569 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:51.795506    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls podName:da984918-11f3-4d23-9381-bd3cc174771f nodeName:}" failed. No retries permitted until 2026-09-16 18:17:07.79549356 +0000 UTC m=+65.660397585 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls") pod "image-registry-84f6855565-qv2zc" (UID: "da984918-11f3-4d23-9381-bd3cc174771f") : secret "image-registry-tls" not found
Sep 16 18:16:51.795569 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:51.795088    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:16:51.795670 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:51.795599    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs podName:67a74600-f466-4ae4-b4bf-67ab02745509 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:07.795587211 +0000 UTC m=+65.660491238 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs") pod "router-default-c65ff548d-bxgzt" (UID: "67a74600-f466-4ae4-b4bf-67ab02745509") : secret "router-metrics-certs-default" not found
Sep 16 18:16:51.795670 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:51.795117    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:16:51.795670 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:51.795654    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls podName:06266e44-9985-4fdc-af4f-1476cabba0fd nodeName:}" failed. No retries permitted until 2026-09-16 18:17:07.795644486 +0000 UTC m=+65.660548516 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-frd2q" (UID: "06266e44-9985-4fdc-af4f-1476cabba0fd") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:16:51.796951 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:51.796927    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:51.797785 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:51.797765    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3828f83-14bf-4688-ba93-b65e204a4118-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-mjv99\" (UID: \"d3828f83-14bf-4688-ba93-b65e204a4118\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"
Sep 16 18:16:51.896047 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:51.896007    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:16:51.896200 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:51.896154    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:16:51.896260 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:51.896216    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls podName:1b6340b4-d303-4081-b1c6-c12e28cb1a31 nodeName:}" failed. No retries permitted until 2026-09-16 18:17:07.896201187 +0000 UTC m=+65.761105213 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls") pod "dns-default-bcg6p" (UID: "1b6340b4-d303-4081-b1c6-c12e28cb1a31") : secret "dns-default-metrics-tls" not found
Sep 16 18:16:51.896571 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:51.896529    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"
Sep 16 18:16:51.922574 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:51.922515    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:52.076401 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:52.076308    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:52.195911 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:52.195880    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:52.227598 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:52.227575    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99"]
Sep 16 18:16:52.447270 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:52.447176    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:52.861515 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:52.861476    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99" event={"ID":"d3828f83-14bf-4688-ba93-b65e204a4118","Type":"ContainerStarted","Data":"72fc3de0a8e1c92ce5b7dfa3ddcc0712164e3ff25a089c7c6d771cd9846b29c0"}
Sep 16 18:16:53.391561 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:53.391508    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:53.568969 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:53.568935    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:53.761216 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:53.761184    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:53.967122 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:53.967070    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:54.116151 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:54.116056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:54.116325 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:54.116262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:54.116430 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:54.116413    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:16:54.116512 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:16:54.116492    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls podName:da9b816c-32e3-4d22-8940-806ebf564ffe nodeName:}" failed. No retries permitted until 2026-09-16 18:17:02.116472518 +0000 UTC m=+59.981376558 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p4k2k" (UID: "da9b816c-32e3-4d22-8940-806ebf564ffe") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:16:54.116760 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:54.116740    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da9b816c-32e3-4d22-8940-806ebf564ffe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:16:54.163747 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:54.163700    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:54.765723 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:54.765630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:54.871731 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:54.871693    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99" event={"ID":"d3828f83-14bf-4688-ba93-b65e204a4118","Type":"ContainerStarted","Data":"266c38cf81c29d3e47905e1980c0470cc56d0f713afb56303b6890e3973ca0dc"}
Sep 16 18:16:54.871890 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:54.871773    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99" event={"ID":"d3828f83-14bf-4688-ba93-b65e204a4118","Type":"ContainerStarted","Data":"6f63a169f8a920ff137276656f181366c15886f512eae32ab07bbd0c7011f1db"}
Sep 16 18:16:54.893387 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:54.893333    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-mjv99" podStartSLOduration=20.708753881 podStartE2EDuration="22.893317735s" totalImagesPullingTime="2.184563854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:54.89295883 +0000 UTC m=+52.757862879" watchObservedRunningTime="2026-09-16 18:16:54.893317735 +0000 UTC m=+52.758221777"
Sep 16 18:16:55.023580 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:55.023468    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:55.163217 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:55.163183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:55.303905 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:55.303822    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:55.403485 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:55.403446    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:55.565106 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:55.565025    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:55.855132 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:55.855043    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:55.978135 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:55.978100    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:56.100022 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:56.099986    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:56.343201 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:56.343158    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:56.343460 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:56.343225    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:56.343588 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:56.343573    2562 scope.go:117] "RemoveContainer" containerID="124b932be39c7160afa11dd128947196ef5e03833c2e34febf97ce7790749b68"
Sep 16 18:16:56.576187 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:56.576156    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:56.880505 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:56.880475    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:16:56.880677 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:56.880533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-mxm76" event={"ID":"e9b403df-9c62-4d05-b8cc-2d2e3d94f521","Type":"ContainerStarted","Data":"a6fc373ac7d01c2fafb278a1013dc315c50b808d063cdd8e7d6c4e785ef51c08"}
Sep 16 18:16:56.880852 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:56.880831    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:56.928137 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:56.928074    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-mxm76" podStartSLOduration=17.171727746 podStartE2EDuration="23.928051284s" totalImagesPullingTime="6.756323538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:56.927356646 +0000 UTC m=+54.792260696" watchObservedRunningTime="2026-09-16 18:16:56.928051284 +0000 UTC m=+54.792955333"
Sep 16 18:16:57.119631 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:57.119597    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-mxm76"
Sep 16 18:16:57.183833 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:57.183753    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:57.367142 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:57.367108    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:57.467381 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:57.467347    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:57.569438 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:57.569407    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:57.674580 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:57.674527    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:57.775011 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:57.774925    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:57.875779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:57.875742    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:57.994639 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:57.994580    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:58.643665 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:58.643630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:58.793697 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:58.793660    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:58.979689 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:58.979653    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:59.076102 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:59.076074    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:59.206816 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:59.206769    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:59.364586 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:59.364488    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:59.460832 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:59.460798    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:59.560623 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:59.560587    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:59.629439 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:59.629349    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:59.764018 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:59.763986    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:59.966469 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:16:59.966431    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:00.555554 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:00.555496    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:00.988114 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:00.988071    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:01.174320 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:01.174271    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:01.363071 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:01.362978    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:01.560683 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:01.560648    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:01.762977 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:01.762950    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4wwpr"
Sep 16 18:17:01.969776 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:01.969740    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:02.189894 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:02.189806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:17:02.192175 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:02.192150    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9b816c-32e3-4d22-8940-806ebf564ffe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p4k2k\" (UID: \"da9b816c-32e3-4d22-8940-806ebf564ffe\") " pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:17:02.209101 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:02.209068    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p4k2k"
Sep 16 18:17:02.358109 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:02.358066    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:02.486025 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:02.485995    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p4k2k"]
Sep 16 18:17:02.489240 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:17:02.489207    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9b816c_32e3_4d22_8940_806ebf564ffe.slice/crio-a4b5a9212d632197ea89c734aec5c020fb476d5e7e4f517c4d9ef925bafa4e3d WatchSource:0}: Error finding container a4b5a9212d632197ea89c734aec5c020fb476d5e7e4f517c4d9ef925bafa4e3d: Status 404 returned error can't find the container with id a4b5a9212d632197ea89c734aec5c020fb476d5e7e4f517c4d9ef925bafa4e3d
Sep 16 18:17:02.566644 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:02.566586    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:02.903578 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:02.903459    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p4k2k" event={"ID":"da9b816c-32e3-4d22-8940-806ebf564ffe","Type":"ContainerStarted","Data":"bcd3da4737a676008701cd1bfe760efb8f6f81fa83d09c9f06974004dcce6bf1"}
Sep 16 18:17:02.903578 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:02.903562    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p4k2k" event={"ID":"da9b816c-32e3-4d22-8940-806ebf564ffe","Type":"ContainerStarted","Data":"a4b5a9212d632197ea89c734aec5c020fb476d5e7e4f517c4d9ef925bafa4e3d"}
Sep 16 18:17:03.908303 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:03.908267    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p4k2k" event={"ID":"da9b816c-32e3-4d22-8940-806ebf564ffe","Type":"ContainerStarted","Data":"2b619a36c05cbab9aea2b3ad06fb59386380208d3f54ff8c5166da7e4a36f721"}
Sep 16 18:17:04.817466 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:04.817424    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:04.913650 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:04.913614    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p4k2k" event={"ID":"da9b816c-32e3-4d22-8940-806ebf564ffe","Type":"ContainerStarted","Data":"e5e6dd0139ff51dce7427d6a62c48e7ddf7ea967223bafd82da7bd7100a0ab00"}
Sep 16 18:17:04.939359 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:04.939303    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p4k2k" podStartSLOduration=16.830806504999998 podStartE2EDuration="18.939286215s" totalImagesPullingTime="2.10847971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:04.937900074 +0000 UTC m=+62.802804133" watchObservedRunningTime="2026-09-16 18:17:04.939286215 +0000 UTC m=+62.804190280"
Sep 16 18:17:07.334531 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.334489    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs\") pod \"network-metrics-daemon-s9p65\" (UID: \"d93e613d-563c-4b51-93c6-4b62fb3d39aa\") " pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:17:07.334981 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.334576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:17:07.337535 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.337504    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:17:07.338209 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.338190    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:17:07.347348 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.347309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a30f7edc-614c-48bc-9559-c7683cc316c0-original-pull-secret\") pod \"global-pull-secret-syncer-ks9dh\" (UID: \"a30f7edc-614c-48bc-9559-c7683cc316c0\") " pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:17:07.347348 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.347327    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d93e613d-563c-4b51-93c6-4b62fb3d39aa-metrics-certs\") pod \"network-metrics-daemon-s9p65\" (UID: \"d93e613d-563c-4b51-93c6-4b62fb3d39aa\") " pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:17:07.435795 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.435751    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89l5d\" (UniqueName: \"kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d\") pod \"network-check-target-rb6xd\" (UID: \"773c8899-dcd1-4682-9732-7761f617a2a5\") " pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:17:07.438290 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.438259    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89l5d\" (UniqueName: \"kubernetes.io/projected/773c8899-dcd1-4682-9732-7761f617a2a5-kube-api-access-89l5d\") pod \"network-check-target-rb6xd\" (UID: \"773c8899-dcd1-4682-9732-7761f617a2a5\") " pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:17:07.456735 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.456708    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xdwg2\""
Sep 16 18:17:07.464777 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.464757    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9p65"
Sep 16 18:17:07.557911 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.557882    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gs7dj\""
Sep 16 18:17:07.562066 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.562049    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks9dh"
Sep 16 18:17:07.565947 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.565932    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:17:07.734297 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.734271    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s9p65"]
Sep 16 18:17:07.736421 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:17:07.736392    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93e613d_563c_4b51_93c6_4b62fb3d39aa.slice/crio-8e07c8e35ddf7d8119d3b151690c62a7428dcb14070f8ef74a7a452440a26059 WatchSource:0}: Error finding container 8e07c8e35ddf7d8119d3b151690c62a7428dcb14070f8ef74a7a452440a26059: Status 404 returned error can't find the container with id 8e07c8e35ddf7d8119d3b151690c62a7428dcb14070f8ef74a7a452440a26059
Sep 16 18:17:07.832446 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.832417    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ks9dh"]
Sep 16 18:17:07.834822 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:17:07.834791    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30f7edc_614c_48bc_9559_c7683cc316c0.slice/crio-92bc29b4fdf05ddb2ebd9759314f78bcb9c17b7dc33f4f6a71086ccb9bd0b6e3 WatchSource:0}: Error finding container 92bc29b4fdf05ddb2ebd9759314f78bcb9c17b7dc33f4f6a71086ccb9bd0b6e3: Status 404 returned error can't find the container with id 92bc29b4fdf05ddb2ebd9759314f78bcb9c17b7dc33f4f6a71086ccb9bd0b6e3
Sep 16 18:17:07.838552 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.838515    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:17:07.838642 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.838620    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:17:07.838705 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.838670    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:17:07.838705 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.838700    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-frd2q\" (UID: \"06266e44-9985-4fdc-af4f-1476cabba0fd\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:17:07.839260 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.839234    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a74600-f466-4ae4-b4bf-67ab02745509-service-ca-bundle\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:17:07.841152 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.841128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a74600-f466-4ae4-b4bf-67ab02745509-metrics-certs\") pod \"router-default-c65ff548d-bxgzt\" (UID: \"67a74600-f466-4ae4-b4bf-67ab02745509\") " pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:17:07.841272 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.841250    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06266e44-9985-4fdc-af4f-1476cabba0fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-frd2q\" (UID: \"06266e44-9985-4fdc-af4f-1476cabba0fd\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:17:07.841432 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.841377    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls\") pod \"image-registry-84f6855565-qv2zc\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") " pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:17:07.842706 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.842659    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rb6xd"]
Sep 16 18:17:07.845141 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:17:07.845118    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773c8899_dcd1_4682_9732_7761f617a2a5.slice/crio-5f452f5c258e978221ecb166d442e7692cfee82e79d2e2b86346d87cc2de91eb WatchSource:0}: Error finding container 5f452f5c258e978221ecb166d442e7692cfee82e79d2e2b86346d87cc2de91eb: Status 404 returned error can't find the container with id 5f452f5c258e978221ecb166d442e7692cfee82e79d2e2b86346d87cc2de91eb
Sep 16 18:17:07.924496 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.924454    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ks9dh" event={"ID":"a30f7edc-614c-48bc-9559-c7683cc316c0","Type":"ContainerStarted","Data":"92bc29b4fdf05ddb2ebd9759314f78bcb9c17b7dc33f4f6a71086ccb9bd0b6e3"}
Sep 16 18:17:07.925715 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.925689    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s9p65" event={"ID":"d93e613d-563c-4b51-93c6-4b62fb3d39aa","Type":"ContainerStarted","Data":"8e07c8e35ddf7d8119d3b151690c62a7428dcb14070f8ef74a7a452440a26059"}
Sep 16 18:17:07.927006 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.926979    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rb6xd" event={"ID":"773c8899-dcd1-4682-9732-7761f617a2a5","Type":"ContainerStarted","Data":"101c209b0e1182e51d5e25e364d0338d7213d0eb57098ea71b3398286bfb7609"}
Sep 16 18:17:07.927118 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.927013    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rb6xd" event={"ID":"773c8899-dcd1-4682-9732-7761f617a2a5","Type":"ContainerStarted","Data":"5f452f5c258e978221ecb166d442e7692cfee82e79d2e2b86346d87cc2de91eb"}
Sep 16 18:17:07.927118 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.927110    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:17:07.939819 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.939796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:17:07.942133 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.942116    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b6340b4-d303-4081-b1c6-c12e28cb1a31-metrics-tls\") pod \"dns-default-bcg6p\" (UID: \"1b6340b4-d303-4081-b1c6-c12e28cb1a31\") " pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:17:07.947241 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:07.947201    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rb6xd" podStartSLOduration=65.947188353 podStartE2EDuration="1m5.947188353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:07.947001037 +0000 UTC m=+65.811905086" watchObservedRunningTime="2026-09-16 18:17:07.947188353 +0000 UTC m=+65.812092401"
Sep 16 18:17:08.092764 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.092726    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8b29l\""
Sep 16 18:17:08.101011 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.100974    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:17:08.107605 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.107584    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gkkf5\""
Sep 16 18:17:08.116023 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.115985    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:17:08.131825 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.131794    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7w9c7\""
Sep 16 18:17:08.140199 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.140165    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"
Sep 16 18:17:08.153954 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.153647    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pfff7\""
Sep 16 18:17:08.161143 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.161061    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:17:08.383683 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.383611    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84f6855565-qv2zc"]
Sep 16 18:17:08.386623 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:17:08.386591    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda984918_11f3_4d23_9381_bd3cc174771f.slice/crio-b35c97ce1ef9009737bd5a2c6efa8c0b548f53ee15722761fca7004ae8162e5f WatchSource:0}: Error finding container b35c97ce1ef9009737bd5a2c6efa8c0b548f53ee15722761fca7004ae8162e5f: Status 404 returned error can't find the container with id b35c97ce1ef9009737bd5a2c6efa8c0b548f53ee15722761fca7004ae8162e5f
Sep 16 18:17:08.410372 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.410300    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c65ff548d-bxgzt"]
Sep 16 18:17:08.439357 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.439332    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q"]
Sep 16 18:17:08.443477 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:17:08.443308    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06266e44_9985_4fdc_af4f_1476cabba0fd.slice/crio-1388e496d14c418a6b6884b757df5b102705ca11161b76288f1018eab6b58e4a WatchSource:0}: Error finding container 1388e496d14c418a6b6884b757df5b102705ca11161b76288f1018eab6b58e4a: Status 404 returned error can't find the container with id 1388e496d14c418a6b6884b757df5b102705ca11161b76288f1018eab6b58e4a
Sep 16 18:17:08.473917 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.473837    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bcg6p"]
Sep 16 18:17:08.479060 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:17:08.478995    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6340b4_d303_4081_b1c6_c12e28cb1a31.slice/crio-02dbe7b1670db3a44b69a00600daea1dd382da0c505d783335179e39c5b6ab3a WatchSource:0}: Error finding container 02dbe7b1670db3a44b69a00600daea1dd382da0c505d783335179e39c5b6ab3a: Status 404 returned error can't find the container with id 02dbe7b1670db3a44b69a00600daea1dd382da0c505d783335179e39c5b6ab3a
Sep 16 18:17:08.936011 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.935914    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s9p65" event={"ID":"d93e613d-563c-4b51-93c6-4b62fb3d39aa","Type":"ContainerStarted","Data":"6a102fe6d6dbcfced59832f2ea753df5d05f25debe9a1c7fac9835e28df3db99"}
Sep 16 18:17:08.938173 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.938133    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bcg6p" event={"ID":"1b6340b4-d303-4081-b1c6-c12e28cb1a31","Type":"ContainerStarted","Data":"02dbe7b1670db3a44b69a00600daea1dd382da0c505d783335179e39c5b6ab3a"}
Sep 16 18:17:08.939800 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.939767    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q" event={"ID":"06266e44-9985-4fdc-af4f-1476cabba0fd","Type":"ContainerStarted","Data":"1388e496d14c418a6b6884b757df5b102705ca11161b76288f1018eab6b58e4a"}
Sep 16 18:17:08.942270 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.942234    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c65ff548d-bxgzt" event={"ID":"67a74600-f466-4ae4-b4bf-67ab02745509","Type":"ContainerStarted","Data":"06322a31d3c62c0f59e22f944c3cd2bad52c03c1a4d942eb632db31de72c0aa7"}
Sep 16 18:17:08.942462 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.942293    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c65ff548d-bxgzt" event={"ID":"67a74600-f466-4ae4-b4bf-67ab02745509","Type":"ContainerStarted","Data":"c528face13be44bf1eda6d85658717ecdf68092ed21aa7f7bdd3a285c42b06ef"}
Sep 16 18:17:08.946379 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.946296    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f6855565-qv2zc" event={"ID":"da984918-11f3-4d23-9381-bd3cc174771f","Type":"ContainerStarted","Data":"0577ea3a14f9f79b99023b00121ceb82c45399bff0e20bf03dcd96ed74537ba5"}
Sep 16 18:17:08.946521 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.946401    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f6855565-qv2zc" event={"ID":"da984918-11f3-4d23-9381-bd3cc174771f","Type":"ContainerStarted","Data":"b35c97ce1ef9009737bd5a2c6efa8c0b548f53ee15722761fca7004ae8162e5f"}
Sep 16 18:17:08.968334 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.968267    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-c65ff548d-bxgzt" podStartSLOduration=36.968241246 podStartE2EDuration="36.968241246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:08.967678026 +0000 UTC m=+66.832582077" watchObservedRunningTime="2026-09-16 18:17:08.968241246 +0000 UTC m=+66.833145295"
Sep 16 18:17:08.990925 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:08.990865    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84f6855565-qv2zc" podStartSLOduration=65.990840873 podStartE2EDuration="1m5.990840873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:08.988998528 +0000 UTC m=+66.853902599" watchObservedRunningTime="2026-09-16 18:17:08.990840873 +0000 UTC m=+66.855744919"
Sep 16 18:17:09.117291 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:09.117208    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:17:09.120370 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:09.120332    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:17:09.954069 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:09.953749    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s9p65" event={"ID":"d93e613d-563c-4b51-93c6-4b62fb3d39aa","Type":"ContainerStarted","Data":"f19956390335846b13762be2096b93e6a40b7415b35bf74a7c6120ff0f5a5712"}
Sep 16 18:17:09.954069 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:09.954003    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:17:09.954895 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:09.954850    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:17:09.955582 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:09.955559    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-c65ff548d-bxgzt"
Sep 16 18:17:09.983722 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:09.983653    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s9p65" podStartSLOduration=66.922639067 podStartE2EDuration="1m7.98362614s" totalImagesPullingTime="1.060987073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:09.982074879 +0000 UTC m=+67.846978927" watchObservedRunningTime="2026-09-16 18:17:09.98362614 +0000 UTC m=+67.848530189"
Sep 16 18:17:12.710577 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.710493    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd99cdb9d-smx75"]
Sep 16 18:17:12.729986 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.729955    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd99cdb9d-smx75"
Sep 16 18:17:12.733426 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.733395    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:17:12.733868 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.733843    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:17:12.735371 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.735338    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-l6jpr\""
Sep 16 18:17:12.735607 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.735591    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:17:12.744902 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.744875    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 18:17:12.745257 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.745230    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd99cdb9d-smx75"]
Sep 16 18:17:12.776596 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.776529    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-d8qn2"]
Sep 16 18:17:12.781285 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.781252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7400954-d0ec-4d1a-86f2-b6fb4090e30f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cd99cdb9d-smx75\" (UID: \"a7400954-d0ec-4d1a-86f2-b6fb4090e30f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd99cdb9d-smx75"
Sep 16 18:17:12.781415 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.781327    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwmd\" (UniqueName: \"kubernetes.io/projected/a7400954-d0ec-4d1a-86f2-b6fb4090e30f-kube-api-access-dzwmd\") pod \"managed-serviceaccount-addon-agent-6cd99cdb9d-smx75\" (UID: \"a7400954-d0ec-4d1a-86f2-b6fb4090e30f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd99cdb9d-smx75"
Sep 16 18:17:12.798798 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.798769    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-d8qn2"
Sep 16 18:17:12.799486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.799458    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-d8qn2"]
Sep 16 18:17:12.801202 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.801178    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lgqwf\""
Sep 16 18:17:12.801842 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.801572    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:17:12.801842 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.801787    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:17:12.882403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.882328    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwmd\" (UniqueName: \"kubernetes.io/projected/a7400954-d0ec-4d1a-86f2-b6fb4090e30f-kube-api-access-dzwmd\") pod \"managed-serviceaccount-addon-agent-6cd99cdb9d-smx75\" (UID: \"a7400954-d0ec-4d1a-86f2-b6fb4090e30f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd99cdb9d-smx75"
Sep 16 18:17:12.882403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.882379    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tqtn\" (UniqueName: \"kubernetes.io/projected/fd86c375-6f0b-445f-9df9-e44ebfea9801-kube-api-access-5tqtn\") pod \"downloads-76f99584f4-d8qn2\" (UID: \"fd86c375-6f0b-445f-9df9-e44ebfea9801\") " pod="openshift-console/downloads-76f99584f4-d8qn2"
Sep 16 18:17:12.882625 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.882455    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7400954-d0ec-4d1a-86f2-b6fb4090e30f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cd99cdb9d-smx75\" (UID: \"a7400954-d0ec-4d1a-86f2-b6fb4090e30f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd99cdb9d-smx75"
Sep 16 18:17:12.885148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.885123    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7400954-d0ec-4d1a-86f2-b6fb4090e30f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cd99cdb9d-smx75\" (UID: \"a7400954-d0ec-4d1a-86f2-b6fb4090e30f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd99cdb9d-smx75"
Sep 16 18:17:12.888137 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.888097    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:12.902720 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.902693    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwmd\" (UniqueName: \"kubernetes.io/projected/a7400954-d0ec-4d1a-86f2-b6fb4090e30f-kube-api-access-dzwmd\") pod \"managed-serviceaccount-addon-agent-6cd99cdb9d-smx75\" (UID: \"a7400954-d0ec-4d1a-86f2-b6fb4090e30f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd99cdb9d-smx75"
Sep 16 18:17:12.968293 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.968259    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ks9dh" event={"ID":"a30f7edc-614c-48bc-9559-c7683cc316c0","Type":"ContainerStarted","Data":"17ec468169133114220e8c3eb0444198fdd51bfdb3b2e60a4f1fe15f8c07f0ed"}
Sep 16 18:17:12.970214 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.970187    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bcg6p" event={"ID":"1b6340b4-d303-4081-b1c6-c12e28cb1a31","Type":"ContainerStarted","Data":"1e05848510cc4d7a332acb0c30e4a403fc906212cb07d2dd4d003b84455e65fb"}
Sep 16 18:17:12.970397 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.970236    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bcg6p" event={"ID":"1b6340b4-d303-4081-b1c6-c12e28cb1a31","Type":"ContainerStarted","Data":"fa2d505259eb6c85990829c3a80b4c1f035c069500537b0391848778aa441213"}
Sep 16 18:17:12.970397 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.970267    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:17:12.971986 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.971942    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q" event={"ID":"06266e44-9985-4fdc-af4f-1476cabba0fd","Type":"ContainerStarted","Data":"88d76c01328883f1c0a0233290d74b1417128b58b081d206695d2f3239abcf3d"}
Sep 16 18:17:12.983049 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.983018    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tqtn\" (UniqueName: \"kubernetes.io/projected/fd86c375-6f0b-445f-9df9-e44ebfea9801-kube-api-access-5tqtn\") pod \"downloads-76f99584f4-d8qn2\" (UID: \"fd86c375-6f0b-445f-9df9-e44ebfea9801\") " pod="openshift-console/downloads-76f99584f4-d8qn2"
Sep 16 18:17:12.995897 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.995872    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tqtn\" (UniqueName: \"kubernetes.io/projected/fd86c375-6f0b-445f-9df9-e44ebfea9801-kube-api-access-5tqtn\") pod \"downloads-76f99584f4-d8qn2\" (UID: \"fd86c375-6f0b-445f-9df9-e44ebfea9801\") " pod="openshift-console/downloads-76f99584f4-d8qn2"
Sep 16 18:17:12.998919 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:12.998887    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:13.017111 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.017048    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-frd2q" podStartSLOduration=37.103475374 podStartE2EDuration="41.017028376s" totalImagesPullingTime="3.913553002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:13.016892436 +0000 UTC m=+70.881796486" watchObservedRunningTime="2026-09-16 18:17:13.017028376 +0000 UTC m=+70.881932426"
Sep 16 18:17:13.017319 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.017296    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ks9dh" podStartSLOduration=66.480530908 podStartE2EDuration="1m11.017288493s" totalImagesPullingTime="4.536757585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:12.990522301 +0000 UTC m=+70.855426350" watchObservedRunningTime="2026-09-16 18:17:13.017288493 +0000 UTC m=+70.882192541"
Sep 16 18:17:13.044687 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.044633    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bcg6p" podStartSLOduration=34.16600639 podStartE2EDuration="38.044614468s" totalImagesPullingTime="3.878608078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:13.043737311 +0000 UTC m=+70.908641360" watchObservedRunningTime="2026-09-16 18:17:13.044614468 +0000 UTC m=+70.909518515"
Sep 16 18:17:13.049857 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.049831    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd99cdb9d-smx75"
Sep 16 18:17:13.100669 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.100616    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:13.111586 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.111520    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-d8qn2"
Sep 16 18:17:13.199944 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.199910    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:13.218772 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.218741    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wbngv"]
Sep 16 18:17:13.240492 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.240464    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wbngv"]
Sep 16 18:17:13.240688 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.240623    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wbngv"
Sep 16 18:17:13.243834 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.243809    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:17:13.243986 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.243930    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-d48sn\""
Sep 16 18:17:13.301160 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.301107    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:13.356348 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.356323    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd99cdb9d-smx75"]
Sep 16 18:17:13.357973 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:17:13.357946    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7400954_d0ec_4d1a_86f2_b6fb4090e30f.slice/crio-836d5b54158eabb1f977f767c273c4ecf8b0c9478e7cd510008c36d2edab6d57 WatchSource:0}: Error finding container 836d5b54158eabb1f977f767c273c4ecf8b0c9478e7cd510008c36d2edab6d57: Status 404 returned error can't find the container with id 836d5b54158eabb1f977f767c273c4ecf8b0c9478e7cd510008c36d2edab6d57
Sep 16 18:17:13.386403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.386373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/58ad7f7a-8727-4c44-8b62-745b724d70fc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-wbngv\" (UID: \"58ad7f7a-8727-4c44-8b62-745b724d70fc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wbngv"
Sep 16 18:17:13.395187 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.395110    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:13.419867 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.419826    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-d8qn2"]
Sep 16 18:17:13.424401 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:17:13.424371    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd86c375_6f0b_445f_9df9_e44ebfea9801.slice/crio-b2c9555bacbd276019fa00c2b4642274233025ca6ffaa2955f36ddeff0599421 WatchSource:0}: Error finding container b2c9555bacbd276019fa00c2b4642274233025ca6ffaa2955f36ddeff0599421: Status 404 returned error can't find the container with id b2c9555bacbd276019fa00c2b4642274233025ca6ffaa2955f36ddeff0599421
Sep 16 18:17:13.486947 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.486913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/58ad7f7a-8727-4c44-8b62-745b724d70fc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-wbngv\" (UID: \"58ad7f7a-8727-4c44-8b62-745b724d70fc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wbngv"
Sep 16 18:17:13.489568 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.489519    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/58ad7f7a-8727-4c44-8b62-745b724d70fc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-wbngv\" (UID: \"58ad7f7a-8727-4c44-8b62-745b724d70fc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wbngv"
Sep 16 18:17:13.551445 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.551410    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wbngv"
Sep 16 18:17:13.922830 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:17:13.922796    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ad7f7a_8727_4c44_8b62_745b724d70fc.slice/crio-7565d63eefabc1ba72b69684bbdf8860b7a70908b681346c701efe1d3c70e4bf WatchSource:0}: Error finding container 7565d63eefabc1ba72b69684bbdf8860b7a70908b681346c701efe1d3c70e4bf: Status 404 returned error can't find the container with id 7565d63eefabc1ba72b69684bbdf8860b7a70908b681346c701efe1d3c70e4bf
Sep 16 18:17:13.930672 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.930639    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wbngv"]
Sep 16 18:17:13.977777 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.977741    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd99cdb9d-smx75" event={"ID":"a7400954-d0ec-4d1a-86f2-b6fb4090e30f","Type":"ContainerStarted","Data":"836d5b54158eabb1f977f767c273c4ecf8b0c9478e7cd510008c36d2edab6d57"}
Sep 16 18:17:13.979038 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.979000    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-d8qn2" event={"ID":"fd86c375-6f0b-445f-9df9-e44ebfea9801","Type":"ContainerStarted","Data":"b2c9555bacbd276019fa00c2b4642274233025ca6ffaa2955f36ddeff0599421"}
Sep 16 18:17:13.980258 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:13.980227    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wbngv" event={"ID":"58ad7f7a-8727-4c44-8b62-745b724d70fc","Type":"ContainerStarted","Data":"7565d63eefabc1ba72b69684bbdf8860b7a70908b681346c701efe1d3c70e4bf"}
Sep 16 18:17:14.307484 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:14.307445    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:14.430975 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:14.430887    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:14.551368 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:14.551309    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:14.639589 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:14.639474    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:15.645683 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:15.645594    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:15.751186 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:15.751147    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:17.132847 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:17.132815    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:17.246490 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:17.246347    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:17.347745 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:17.347704    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:17.445957 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:17.445914    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:17.554225 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:17.554132    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:17.935342 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:17.935246    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:18.000858 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:18.000810    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wbngv" event={"ID":"58ad7f7a-8727-4c44-8b62-745b724d70fc","Type":"ContainerStarted","Data":"8b6f720e47260f4cedbc303d181a7ee915d8dbb8806ffcf7437386a611fd94d5"}
Sep 16 18:17:18.001056 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:18.001009    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wbngv"
Sep 16 18:17:18.003077 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:18.003020    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd99cdb9d-smx75" event={"ID":"a7400954-d0ec-4d1a-86f2-b6fb4090e30f","Type":"ContainerStarted","Data":"a42dc12dff2b229b7f4908c78e0f698c72fd4cf4f2e4029b78409d64893b6359"}
Sep 16 18:17:18.006954 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:18.006889    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wbngv"
Sep 16 18:17:18.014099 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:18.014059    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:18.025683 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:18.025620    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-wbngv" podStartSLOduration=1.934241334 podStartE2EDuration="5.025601527s" totalImagesPullingTime="3.091360193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:18.023915167 +0000 UTC m=+75.888819215" watchObservedRunningTime="2026-09-16 18:17:18.025601527 +0000 UTC m=+75.890505573"
Sep 16 18:17:18.076509 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:18.076451    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd99cdb9d-smx75" podStartSLOduration=2.423103023 podStartE2EDuration="6.076426375s" totalImagesPullingTime="3.653323352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:18.075236592 +0000 UTC m=+75.940140641" watchObservedRunningTime="2026-09-16 18:17:18.076426375 +0000 UTC m=+75.941330423"
Sep 16 18:17:18.106247 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:18.106214    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:18.236032 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:18.235990    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:18.329886 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:18.329844    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:18.531441 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:18.531352    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:18.622131 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:18.622085    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:18.718438 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:18.718393    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:21.288418 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:21.288383    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:21.413166 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:21.413127    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:22.541932 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:22.541891    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:22.763373 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:22.763333    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:22.960350 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:22.960315    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:22.983368 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:22.983336    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bcg6p"
Sep 16 18:17:23.132690 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:23.132652    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:24.604816 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:24.604780    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:24.740963 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:24.740926    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:25.158320 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.158167    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:25.360018 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.359978    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:25.516987 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.516951    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:25.649153 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.649115    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ftk6p"]
Sep 16 18:17:25.652905 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.652873    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.659875 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.659847    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:17:25.660054 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.659978    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-976fz\""
Sep 16 18:17:25.660054 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.660039    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:17:25.660368 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.660315    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:17:25.660822 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.660802    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:17:25.795391 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.795310    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-textfile\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.795391 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.795355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-wtmp\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.795653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.795399    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.795653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.795425    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/45df9a5b-cf94-45a3-9447-08d3b3db251f-root\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.795653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.795446    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45df9a5b-cf94-45a3-9447-08d3b3db251f-sys\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.795653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.795468    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-accelerators-collector-config\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.795653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.795518    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45df9a5b-cf94-45a3-9447-08d3b3db251f-metrics-client-ca\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.795653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.795576    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-tls\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.795653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.795612    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcjc8\" (UniqueName: \"kubernetes.io/projected/45df9a5b-cf94-45a3-9447-08d3b3db251f-kube-api-access-xcjc8\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.796457 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.796429    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:25.896887 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.896844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-wtmp\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.897093 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.896913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.897093 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.896954    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/45df9a5b-cf94-45a3-9447-08d3b3db251f-root\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.897093 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.896983    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45df9a5b-cf94-45a3-9447-08d3b3db251f-sys\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.897093 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.897005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-accelerators-collector-config\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.897093 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.897036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45df9a5b-cf94-45a3-9447-08d3b3db251f-metrics-client-ca\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.897093 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.897058    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-wtmp\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.897093 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.897074    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/45df9a5b-cf94-45a3-9447-08d3b3db251f-root\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.897093 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.897065    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-tls\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.897473 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.897074    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45df9a5b-cf94-45a3-9447-08d3b3db251f-sys\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.897473 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.897122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcjc8\" (UniqueName: \"kubernetes.io/projected/45df9a5b-cf94-45a3-9447-08d3b3db251f-kube-api-access-xcjc8\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.897473 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:17:25.897152    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:17:25.897473 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.897178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-textfile\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.897473 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:17:25.897219    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-tls podName:45df9a5b-cf94-45a3-9447-08d3b3db251f nodeName:}" failed. No retries permitted until 2026-09-16 18:17:26.397197209 +0000 UTC m=+84.262101247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-tls") pod "node-exporter-ftk6p" (UID: "45df9a5b-cf94-45a3-9447-08d3b3db251f") : secret "node-exporter-tls" not found
Sep 16 18:17:25.897804 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.897631    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-textfile\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.897861 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.897797    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-accelerators-collector-config\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.898025 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.898002    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45df9a5b-cf94-45a3-9447-08d3b3db251f-metrics-client-ca\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.900205 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.900180    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.947799 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.947697    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcjc8\" (UniqueName: \"kubernetes.io/projected/45df9a5b-cf94-45a3-9447-08d3b3db251f-kube-api-access-xcjc8\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:25.999620 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:25.999576    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:26.239192 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:26.239133    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:26.403029 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:26.402985    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-tls\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:26.405729 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:26.405699    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45df9a5b-cf94-45a3-9447-08d3b3db251f-node-exporter-tls\") pod \"node-exporter-ftk6p\" (UID: \"45df9a5b-cf94-45a3-9447-08d3b3db251f\") " pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:26.448388 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:26.448338    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:26.565232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:26.565137    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ftk6p"
Sep 16 18:17:26.769444 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:26.769399    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:26.944452 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:26.944351    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:27.228210 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:27.228168    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:27.355676 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:27.355642    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:27.499227 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:27.499129    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:28.851760 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:28.851722    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:28.951340 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:28.951292    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:29.181717 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:29.181607    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:29.318592 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:29.318553    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:29.604698 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:29.604660    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:29.770233 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:29.770195    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:29.977094 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:29.977052    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:30.143394 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:30.143358    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:30.336729 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:17:30.336692    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45df9a5b_cf94_45a3_9447_08d3b3db251f.slice/crio-da0833be4e9bcb0729e8c6bca1fd87d4afae3fb2e10cc1100bdacb989a5743ac WatchSource:0}: Error finding container da0833be4e9bcb0729e8c6bca1fd87d4afae3fb2e10cc1100bdacb989a5743ac: Status 404 returned error can't find the container with id da0833be4e9bcb0729e8c6bca1fd87d4afae3fb2e10cc1100bdacb989a5743ac
Sep 16 18:17:30.542965 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:30.542923    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:30.964690 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:30.964286    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:17:31.066423 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:31.063661    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-d8qn2" event={"ID":"fd86c375-6f0b-445f-9df9-e44ebfea9801","Type":"ContainerStarted","Data":"17c642f2a93d192e3109a175cedc83ea004d1a80b13cd1ce9f84633addc85253"}
Sep 16 18:17:31.066423 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:31.064635    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-d8qn2"
Sep 16 18:17:31.067718 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:31.067678    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftk6p" event={"ID":"45df9a5b-cf94-45a3-9447-08d3b3db251f","Type":"ContainerStarted","Data":"da0833be4e9bcb0729e8c6bca1fd87d4afae3fb2e10cc1100bdacb989a5743ac"}
Sep 16 18:17:31.073195 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:31.073167    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-d8qn2"
Sep 16 18:17:31.147976 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:31.147892    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-d8qn2" podStartSLOduration=2.143272973 podStartE2EDuration="19.147871146s" totalImagesPullingTime="17.004598173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:31.105493962 +0000 UTC m=+88.970398011" watchObservedRunningTime="2026-09-16 18:17:31.147871146 +0000 UTC m=+89.012775194"
Sep 16 18:17:31.345185 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:31.345042    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:31.473313 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:31.473220    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:32.075126 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:32.075087    2562 generic.go:356] "Generic (PLEG): container finished" podID="45df9a5b-cf94-45a3-9447-08d3b3db251f" containerID="0c883b752ec0c4cf8fb10e3057db555fb786179b5336e0d48231df43b185e48b" exitCode=0
Sep 16 18:17:32.075615 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:32.075186    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftk6p" event={"ID":"45df9a5b-cf94-45a3-9447-08d3b3db251f","Type":"ContainerDied","Data":"0c883b752ec0c4cf8fb10e3057db555fb786179b5336e0d48231df43b185e48b"}
Sep 16 18:17:32.303817 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:32.303708    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:32.405311 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:32.405270    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:33.082988 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:33.082944    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftk6p" event={"ID":"45df9a5b-cf94-45a3-9447-08d3b3db251f","Type":"ContainerStarted","Data":"a2cc64acb6b07bac3880cde5590aa57435dee5e5fae08d4a31847fb3ef8939be"}
Sep 16 18:17:33.083471 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:33.083016    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftk6p" event={"ID":"45df9a5b-cf94-45a3-9447-08d3b3db251f","Type":"ContainerStarted","Data":"94d3a220e0fb3a8d0ccfe71d83fd99c91e6b3d51260e5df71f596b98d3247506"}
Sep 16 18:17:33.112117 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:33.112047    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ftk6p" podStartSLOduration=6.900928684 podStartE2EDuration="8.112022516s" totalImagesPullingTime="1.132304446s" totalInitContainerRuntime="78.789386ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:17:33.109570024 +0000 UTC m=+90.974474074" watchObservedRunningTime="2026-09-16 18:17:33.112022516 +0000 UTC m=+90.976926568"
Sep 16 18:17:36.301826 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:36.301789    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84f6855565-qv2zc"]
Sep 16 18:17:36.407853 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:36.407808    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:36.530695 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:36.530650    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:38.949456 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:38.949420    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rb6xd"
Sep 16 18:17:39.071911 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:39.071867    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:39.176820 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:39.176777    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:39.251826 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:39.251713    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:41.257653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:41.257597    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:41.346591 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:41.346534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:44.564651 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:44.564612    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:44.655607 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:44.655571    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:45.706824 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:45.706792    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:45.807406 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:45.807371    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:45.887785 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:45.887749    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:56.172161 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:56.172116    2562 generic.go:356] "Generic (PLEG): container finished" podID="4642a843-fe4e-43a4-ab7e-64cc1c66bf10" containerID="94ef66c98026b501afe377ad7283e64354fa1249df8d1cf040920a6d560201c6" exitCode=0
Sep 16 18:17:56.172580 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:56.172185    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wnqkf" event={"ID":"4642a843-fe4e-43a4-ab7e-64cc1c66bf10","Type":"ContainerDied","Data":"94ef66c98026b501afe377ad7283e64354fa1249df8d1cf040920a6d560201c6"}
Sep 16 18:17:56.172580 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:56.172499    2562 scope.go:117] "RemoveContainer" containerID="94ef66c98026b501afe377ad7283e64354fa1249df8d1cf040920a6d560201c6"
Sep 16 18:17:57.177380 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:57.177343    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wnqkf" event={"ID":"4642a843-fe4e-43a4-ab7e-64cc1c66bf10","Type":"ContainerStarted","Data":"5e06398ad0a200ef4888bc20644ed5f3a6ad537b7175eda65fcebb0728f03f68"}
Sep 16 18:17:59.555416 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:59.555370    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:59.634452 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:17:59.634418    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:01.333923 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.333883    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84f6855565-qv2zc" podUID="da984918-11f3-4d23-9381-bd3cc174771f" containerName="registry" containerID="cri-o://0577ea3a14f9f79b99023b00121ceb82c45399bff0e20bf03dcd96ed74537ba5" gracePeriod=30
Sep 16 18:18:01.604306 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.604234    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:18:01.686583 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.686551    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da984918-11f3-4d23-9381-bd3cc174771f-registry-certificates\") pod \"da984918-11f3-4d23-9381-bd3cc174771f\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") "
Sep 16 18:18:01.686762 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.686590    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd5fl\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-kube-api-access-vd5fl\") pod \"da984918-11f3-4d23-9381-bd3cc174771f\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") "
Sep 16 18:18:01.686762 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.686611    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da984918-11f3-4d23-9381-bd3cc174771f-image-registry-private-configuration\") pod \"da984918-11f3-4d23-9381-bd3cc174771f\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") "
Sep 16 18:18:01.686762 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.686663    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-bound-sa-token\") pod \"da984918-11f3-4d23-9381-bd3cc174771f\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") "
Sep 16 18:18:01.686762 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.686678    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls\") pod \"da984918-11f3-4d23-9381-bd3cc174771f\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") "
Sep 16 18:18:01.686762 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.686704    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da984918-11f3-4d23-9381-bd3cc174771f-trusted-ca\") pod \"da984918-11f3-4d23-9381-bd3cc174771f\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") "
Sep 16 18:18:01.686762 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.686749    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da984918-11f3-4d23-9381-bd3cc174771f-ca-trust-extracted\") pod \"da984918-11f3-4d23-9381-bd3cc174771f\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") "
Sep 16 18:18:01.687071 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.686774    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da984918-11f3-4d23-9381-bd3cc174771f-installation-pull-secrets\") pod \"da984918-11f3-4d23-9381-bd3cc174771f\" (UID: \"da984918-11f3-4d23-9381-bd3cc174771f\") "
Sep 16 18:18:01.687349 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.687314    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da984918-11f3-4d23-9381-bd3cc174771f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "da984918-11f3-4d23-9381-bd3cc174771f" (UID: "da984918-11f3-4d23-9381-bd3cc174771f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:18:01.687449 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.687349    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da984918-11f3-4d23-9381-bd3cc174771f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "da984918-11f3-4d23-9381-bd3cc174771f" (UID: "da984918-11f3-4d23-9381-bd3cc174771f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:18:01.689624 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.689587    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "da984918-11f3-4d23-9381-bd3cc174771f" (UID: "da984918-11f3-4d23-9381-bd3cc174771f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:18:01.689758 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.689672    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-kube-api-access-vd5fl" (OuterVolumeSpecName: "kube-api-access-vd5fl") pod "da984918-11f3-4d23-9381-bd3cc174771f" (UID: "da984918-11f3-4d23-9381-bd3cc174771f"). InnerVolumeSpecName "kube-api-access-vd5fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:18:01.689758 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.689720    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "da984918-11f3-4d23-9381-bd3cc174771f" (UID: "da984918-11f3-4d23-9381-bd3cc174771f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:18:01.689859 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.689775    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da984918-11f3-4d23-9381-bd3cc174771f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "da984918-11f3-4d23-9381-bd3cc174771f" (UID: "da984918-11f3-4d23-9381-bd3cc174771f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:18:01.689894 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.689850    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da984918-11f3-4d23-9381-bd3cc174771f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "da984918-11f3-4d23-9381-bd3cc174771f" (UID: "da984918-11f3-4d23-9381-bd3cc174771f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:18:01.696504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.696474    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da984918-11f3-4d23-9381-bd3cc174771f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "da984918-11f3-4d23-9381-bd3cc174771f" (UID: "da984918-11f3-4d23-9381-bd3cc174771f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:18:01.787840 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.787809    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-bound-sa-token\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:18:01.787840 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.787835    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-registry-tls\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:18:01.787840 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.787844    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/da984918-11f3-4d23-9381-bd3cc174771f-trusted-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:18:01.788050 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.787852    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/da984918-11f3-4d23-9381-bd3cc174771f-ca-trust-extracted\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:18:01.788050 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.787862    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/da984918-11f3-4d23-9381-bd3cc174771f-installation-pull-secrets\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:18:01.788050 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.787871    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/da984918-11f3-4d23-9381-bd3cc174771f-registry-certificates\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:18:01.788050 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.787880    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd5fl\" (UniqueName: \"kubernetes.io/projected/da984918-11f3-4d23-9381-bd3cc174771f-kube-api-access-vd5fl\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:18:01.788050 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:01.787889    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/da984918-11f3-4d23-9381-bd3cc174771f-image-registry-private-configuration\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:18:02.196121 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:02.196083    2562 generic.go:356] "Generic (PLEG): container finished" podID="da984918-11f3-4d23-9381-bd3cc174771f" containerID="0577ea3a14f9f79b99023b00121ceb82c45399bff0e20bf03dcd96ed74537ba5" exitCode=0
Sep 16 18:18:02.196287 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:02.196146    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f6855565-qv2zc"
Sep 16 18:18:02.196287 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:02.196160    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f6855565-qv2zc" event={"ID":"da984918-11f3-4d23-9381-bd3cc174771f","Type":"ContainerDied","Data":"0577ea3a14f9f79b99023b00121ceb82c45399bff0e20bf03dcd96ed74537ba5"}
Sep 16 18:18:02.196287 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:02.196231    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f6855565-qv2zc" event={"ID":"da984918-11f3-4d23-9381-bd3cc174771f","Type":"ContainerDied","Data":"b35c97ce1ef9009737bd5a2c6efa8c0b548f53ee15722761fca7004ae8162e5f"}
Sep 16 18:18:02.196287 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:02.196255    2562 scope.go:117] "RemoveContainer" containerID="0577ea3a14f9f79b99023b00121ceb82c45399bff0e20bf03dcd96ed74537ba5"
Sep 16 18:18:02.204992 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:02.204974    2562 scope.go:117] "RemoveContainer" containerID="0577ea3a14f9f79b99023b00121ceb82c45399bff0e20bf03dcd96ed74537ba5"
Sep 16 18:18:02.205272 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:18:02.205252    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0577ea3a14f9f79b99023b00121ceb82c45399bff0e20bf03dcd96ed74537ba5\": container with ID starting with 0577ea3a14f9f79b99023b00121ceb82c45399bff0e20bf03dcd96ed74537ba5 not found: ID does not exist" containerID="0577ea3a14f9f79b99023b00121ceb82c45399bff0e20bf03dcd96ed74537ba5"
Sep 16 18:18:02.205338 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:02.205283    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0577ea3a14f9f79b99023b00121ceb82c45399bff0e20bf03dcd96ed74537ba5"} err="failed to get container status \"0577ea3a14f9f79b99023b00121ceb82c45399bff0e20bf03dcd96ed74537ba5\": rpc error: code = NotFound desc = could not find container \"0577ea3a14f9f79b99023b00121ceb82c45399bff0e20bf03dcd96ed74537ba5\": container with ID starting with 0577ea3a14f9f79b99023b00121ceb82c45399bff0e20bf03dcd96ed74537ba5 not found: ID does not exist"
Sep 16 18:18:02.218016 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:02.217979    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84f6855565-qv2zc"]
Sep 16 18:18:02.223318 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:02.223291    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84f6855565-qv2zc"]
Sep 16 18:18:02.548425 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:02.548395    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da984918-11f3-4d23-9381-bd3cc174771f" path="/var/lib/kubelet/pods/da984918-11f3-4d23-9381-bd3cc174771f/volumes"
Sep 16 18:18:11.233227 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:11.233189    2562 generic.go:356] "Generic (PLEG): container finished" podID="4a69f057-7c16-42af-8e69-9be2c06c02c7" containerID="2a22163433e1e02bf20a4a73d3f16f81ef014465aa312775d25e4023275c847c" exitCode=0
Sep 16 18:18:11.233757 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:11.233232    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4" event={"ID":"4a69f057-7c16-42af-8e69-9be2c06c02c7","Type":"ContainerDied","Data":"2a22163433e1e02bf20a4a73d3f16f81ef014465aa312775d25e4023275c847c"}
Sep 16 18:18:11.233757 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:11.233625    2562 scope.go:117] "RemoveContainer" containerID="2a22163433e1e02bf20a4a73d3f16f81ef014465aa312775d25e4023275c847c"
Sep 16 18:18:12.238596 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:12.238536    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-wrll4" event={"ID":"4a69f057-7c16-42af-8e69-9be2c06c02c7","Type":"ContainerStarted","Data":"4ded727b6dac3373e602b183712780baadc43053c1f9a2c7d2b2a6d8d042d250"}
Sep 16 18:18:22.687498 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:22.687447    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:26.175832 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:26.175768    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:26.250496 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:26.250452    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:40.550031 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:18:40.549990    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:19.680537 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:19:19.680496    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:26.927996 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:19:26.927941    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:02.486732 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:21:02.486699    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:21:02.492256 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:21:02.492229    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:21:02.493405 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:21:02.493384    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:21:02.496204 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:21:02.496182    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:21:02.499617 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:21:02.499598    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:22:33.636786 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:22:33.636745    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:22:35.144682 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:22:35.144642    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:22:36.841873 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:22:36.841831    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:23:46.787821 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:23:46.787739    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:23:46.915051 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:23:46.915020    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:23:57.578539 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:23:57.578502    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:23:57.690395 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:23:57.690363    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:23:58.342371 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:23:58.342336    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:23:58.442569 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:23:58.442527    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:23:58.690382 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:23:58.690296    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:23:58.803559 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:23:58.803493    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:01.335303 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:24:01.335262    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:08.126443 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:24:08.126410    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:08.230775 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:24:08.230742    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:08.297496 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:24:08.297462    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:14.556390 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:24:14.556360    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:14.656710 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:24:14.656673    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:37.239509 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:24:37.239468    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:44.855937 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:24:44.855892    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:44.957321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:24:44.957280    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:03.264486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:03.264450    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:03.364967 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:03.364937    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:04.116665 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.116629    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"]
Sep 16 18:25:04.116910 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.116897    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da984918-11f3-4d23-9381-bd3cc174771f" containerName="registry"
Sep 16 18:25:04.116956 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.116910    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da984918-11f3-4d23-9381-bd3cc174771f" containerName="registry"
Sep 16 18:25:04.116990 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.116957    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="da984918-11f3-4d23-9381-bd3cc174771f" containerName="registry"
Sep 16 18:25:04.118798 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.118780    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"
Sep 16 18:25:04.121246 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.121223    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-28c5h\""
Sep 16 18:25:04.121246 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.121241    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:25:04.122313 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.122284    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:25:04.127294 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.127274    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"]
Sep 16 18:25:04.248205 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.248175    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2527a56f-1996-46e4-82e2-f222e7c3e3e4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh\" (UID: \"2527a56f-1996-46e4-82e2-f222e7c3e3e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"
Sep 16 18:25:04.248389 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.248214    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2527a56f-1996-46e4-82e2-f222e7c3e3e4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh\" (UID: \"2527a56f-1996-46e4-82e2-f222e7c3e3e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"
Sep 16 18:25:04.248389 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.248247    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x8k2\" (UniqueName: \"kubernetes.io/projected/2527a56f-1996-46e4-82e2-f222e7c3e3e4-kube-api-access-4x8k2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh\" (UID: \"2527a56f-1996-46e4-82e2-f222e7c3e3e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"
Sep 16 18:25:04.349263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.349231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2527a56f-1996-46e4-82e2-f222e7c3e3e4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh\" (UID: \"2527a56f-1996-46e4-82e2-f222e7c3e3e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"
Sep 16 18:25:04.349734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.349282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2527a56f-1996-46e4-82e2-f222e7c3e3e4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh\" (UID: \"2527a56f-1996-46e4-82e2-f222e7c3e3e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"
Sep 16 18:25:04.349734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.349315    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x8k2\" (UniqueName: \"kubernetes.io/projected/2527a56f-1996-46e4-82e2-f222e7c3e3e4-kube-api-access-4x8k2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh\" (UID: \"2527a56f-1996-46e4-82e2-f222e7c3e3e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"
Sep 16 18:25:04.349734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.349625    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2527a56f-1996-46e4-82e2-f222e7c3e3e4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh\" (UID: \"2527a56f-1996-46e4-82e2-f222e7c3e3e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"
Sep 16 18:25:04.349734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.349643    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2527a56f-1996-46e4-82e2-f222e7c3e3e4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh\" (UID: \"2527a56f-1996-46e4-82e2-f222e7c3e3e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"
Sep 16 18:25:04.358010 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.357985    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x8k2\" (UniqueName: \"kubernetes.io/projected/2527a56f-1996-46e4-82e2-f222e7c3e3e4-kube-api-access-4x8k2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh\" (UID: \"2527a56f-1996-46e4-82e2-f222e7c3e3e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"
Sep 16 18:25:04.427691 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.427616    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"
Sep 16 18:25:04.696371 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.696345    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"]
Sep 16 18:25:04.698911 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:25:04.698876    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2527a56f_1996_46e4_82e2_f222e7c3e3e4.slice/crio-20d52ebb9282c2c04329c0b01b87df635f1b775a4cc24eacd928faf5fd3f9752 WatchSource:0}: Error finding container 20d52ebb9282c2c04329c0b01b87df635f1b775a4cc24eacd928faf5fd3f9752: Status 404 returned error can't find the container with id 20d52ebb9282c2c04329c0b01b87df635f1b775a4cc24eacd928faf5fd3f9752
Sep 16 18:25:04.700888 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.700869    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:25:04.720067 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:04.720042    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh" event={"ID":"2527a56f-1996-46e4-82e2-f222e7c3e3e4","Type":"ContainerStarted","Data":"20d52ebb9282c2c04329c0b01b87df635f1b775a4cc24eacd928faf5fd3f9752"}
Sep 16 18:25:12.754072 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:12.754034    2562 generic.go:356] "Generic (PLEG): container finished" podID="2527a56f-1996-46e4-82e2-f222e7c3e3e4" containerID="8cf00897fc1ea2e54b68d4988d5be3f5c670ccd17934b5a7b9f7565a55f932eb" exitCode=0
Sep 16 18:25:12.754446 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:12.754115    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh" event={"ID":"2527a56f-1996-46e4-82e2-f222e7c3e3e4","Type":"ContainerDied","Data":"8cf00897fc1ea2e54b68d4988d5be3f5c670ccd17934b5a7b9f7565a55f932eb"}
Sep 16 18:25:15.770417 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:15.770381    2562 generic.go:356] "Generic (PLEG): container finished" podID="2527a56f-1996-46e4-82e2-f222e7c3e3e4" containerID="85e1c72fb972fcc563bcf9a1389cb3eb71028d8c3986da0d0aedacaf58129d61" exitCode=0
Sep 16 18:25:15.770818 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:15.770474    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh" event={"ID":"2527a56f-1996-46e4-82e2-f222e7c3e3e4","Type":"ContainerDied","Data":"85e1c72fb972fcc563bcf9a1389cb3eb71028d8c3986da0d0aedacaf58129d61"}
Sep 16 18:25:22.801934 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:22.801900    2562 generic.go:356] "Generic (PLEG): container finished" podID="2527a56f-1996-46e4-82e2-f222e7c3e3e4" containerID="c2e92d02ddb67d2c6d30968584cad005d83dacfd2ed15bee4a163fb707892fb4" exitCode=0
Sep 16 18:25:22.802320 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:22.801958    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh" event={"ID":"2527a56f-1996-46e4-82e2-f222e7c3e3e4","Type":"ContainerDied","Data":"c2e92d02ddb67d2c6d30968584cad005d83dacfd2ed15bee4a163fb707892fb4"}
Sep 16 18:25:23.928638 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:23.928616    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"
Sep 16 18:25:24.005465 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:24.005431    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2527a56f-1996-46e4-82e2-f222e7c3e3e4-bundle\") pod \"2527a56f-1996-46e4-82e2-f222e7c3e3e4\" (UID: \"2527a56f-1996-46e4-82e2-f222e7c3e3e4\") "
Sep 16 18:25:24.005657 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:24.005472    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x8k2\" (UniqueName: \"kubernetes.io/projected/2527a56f-1996-46e4-82e2-f222e7c3e3e4-kube-api-access-4x8k2\") pod \"2527a56f-1996-46e4-82e2-f222e7c3e3e4\" (UID: \"2527a56f-1996-46e4-82e2-f222e7c3e3e4\") "
Sep 16 18:25:24.005657 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:24.005533    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2527a56f-1996-46e4-82e2-f222e7c3e3e4-util\") pod \"2527a56f-1996-46e4-82e2-f222e7c3e3e4\" (UID: \"2527a56f-1996-46e4-82e2-f222e7c3e3e4\") "
Sep 16 18:25:24.006162 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:24.006132    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2527a56f-1996-46e4-82e2-f222e7c3e3e4-bundle" (OuterVolumeSpecName: "bundle") pod "2527a56f-1996-46e4-82e2-f222e7c3e3e4" (UID: "2527a56f-1996-46e4-82e2-f222e7c3e3e4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:25:24.007676 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:24.007651    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2527a56f-1996-46e4-82e2-f222e7c3e3e4-kube-api-access-4x8k2" (OuterVolumeSpecName: "kube-api-access-4x8k2") pod "2527a56f-1996-46e4-82e2-f222e7c3e3e4" (UID: "2527a56f-1996-46e4-82e2-f222e7c3e3e4"). InnerVolumeSpecName "kube-api-access-4x8k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:25:24.010173 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:24.010151    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2527a56f-1996-46e4-82e2-f222e7c3e3e4-util" (OuterVolumeSpecName: "util") pod "2527a56f-1996-46e4-82e2-f222e7c3e3e4" (UID: "2527a56f-1996-46e4-82e2-f222e7c3e3e4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:25:24.106741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:24.106655    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2527a56f-1996-46e4-82e2-f222e7c3e3e4-util\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:25:24.106741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:24.106687    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2527a56f-1996-46e4-82e2-f222e7c3e3e4-bundle\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:25:24.106741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:24.106696    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x8k2\" (UniqueName: \"kubernetes.io/projected/2527a56f-1996-46e4-82e2-f222e7c3e3e4-kube-api-access-4x8k2\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:25:24.812317 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:24.812227    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh"
Sep 16 18:25:24.812492 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:24.812230    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xk2wh" event={"ID":"2527a56f-1996-46e4-82e2-f222e7c3e3e4","Type":"ContainerDied","Data":"20d52ebb9282c2c04329c0b01b87df635f1b775a4cc24eacd928faf5fd3f9752"}
Sep 16 18:25:24.812492 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:24.812349    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20d52ebb9282c2c04329c0b01b87df635f1b775a4cc24eacd928faf5fd3f9752"
Sep 16 18:25:30.224040 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:30.223987    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:31.213528 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:31.213466    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:32.194666 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.194628    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nkrr4"]
Sep 16 18:25:32.195027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.194890    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2527a56f-1996-46e4-82e2-f222e7c3e3e4" containerName="extract"
Sep 16 18:25:32.195027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.194901    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2527a56f-1996-46e4-82e2-f222e7c3e3e4" containerName="extract"
Sep 16 18:25:32.195027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.194912    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2527a56f-1996-46e4-82e2-f222e7c3e3e4" containerName="util"
Sep 16 18:25:32.195027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.194917    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2527a56f-1996-46e4-82e2-f222e7c3e3e4" containerName="util"
Sep 16 18:25:32.195027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.194925    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2527a56f-1996-46e4-82e2-f222e7c3e3e4" containerName="pull"
Sep 16 18:25:32.195027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.194931    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2527a56f-1996-46e4-82e2-f222e7c3e3e4" containerName="pull"
Sep 16 18:25:32.195027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.194977    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2527a56f-1996-46e4-82e2-f222e7c3e3e4" containerName="extract"
Sep 16 18:25:32.197067 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.197050    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nkrr4"
Sep 16 18:25:32.199649 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.199617    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 18:25:32.199795 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.199770    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:25:32.199918 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.199902    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ttzd9\""
Sep 16 18:25:32.212123 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.212098    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nkrr4"]
Sep 16 18:25:32.259060 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.259027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/870f09c6-50c3-4067-a962-997654665528-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nkrr4\" (UID: \"870f09c6-50c3-4067-a962-997654665528\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nkrr4"
Sep 16 18:25:32.259235 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.259085    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8lx7\" (UniqueName: \"kubernetes.io/projected/870f09c6-50c3-4067-a962-997654665528-kube-api-access-r8lx7\") pod \"cert-manager-operator-controller-manager-55c444bf86-nkrr4\" (UID: \"870f09c6-50c3-4067-a962-997654665528\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nkrr4"
Sep 16 18:25:32.360026 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.359988    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/870f09c6-50c3-4067-a962-997654665528-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nkrr4\" (UID: \"870f09c6-50c3-4067-a962-997654665528\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nkrr4"
Sep 16 18:25:32.360099 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.360064    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lx7\" (UniqueName: \"kubernetes.io/projected/870f09c6-50c3-4067-a962-997654665528-kube-api-access-r8lx7\") pod \"cert-manager-operator-controller-manager-55c444bf86-nkrr4\" (UID: \"870f09c6-50c3-4067-a962-997654665528\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nkrr4"
Sep 16 18:25:32.360359 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.360339    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/870f09c6-50c3-4067-a962-997654665528-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-nkrr4\" (UID: \"870f09c6-50c3-4067-a962-997654665528\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nkrr4"
Sep 16 18:25:32.368441 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.368421    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8lx7\" (UniqueName: \"kubernetes.io/projected/870f09c6-50c3-4067-a962-997654665528-kube-api-access-r8lx7\") pod \"cert-manager-operator-controller-manager-55c444bf86-nkrr4\" (UID: \"870f09c6-50c3-4067-a962-997654665528\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nkrr4"
Sep 16 18:25:32.505664 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.505630    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nkrr4"
Sep 16 18:25:32.788078 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.788053    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nkrr4"]
Sep 16 18:25:32.790832 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:25:32.790801    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870f09c6_50c3_4067_a962_997654665528.slice/crio-f951e947286dcd4feb57e75cf09c14c2470c8e601fca129acb20d7a1fa9cfdc1 WatchSource:0}: Error finding container f951e947286dcd4feb57e75cf09c14c2470c8e601fca129acb20d7a1fa9cfdc1: Status 404 returned error can't find the container with id f951e947286dcd4feb57e75cf09c14c2470c8e601fca129acb20d7a1fa9cfdc1
Sep 16 18:25:32.848348 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:32.848317    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nkrr4" event={"ID":"870f09c6-50c3-4067-a962-997654665528","Type":"ContainerStarted","Data":"f951e947286dcd4feb57e75cf09c14c2470c8e601fca129acb20d7a1fa9cfdc1"}
Sep 16 18:25:33.903900 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:33.903858    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:34.003335 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:34.003304    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:35.727658 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:35.727624    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:35.822070 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:35.822039    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:35.861709 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:35.861672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nkrr4" event={"ID":"870f09c6-50c3-4067-a962-997654665528","Type":"ContainerStarted","Data":"64a244535e116fe133057fac6d5ae8a8ef119cde6272b173b82299b9dedeec7c"}
Sep 16 18:25:35.888417 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:35.888364    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-nkrr4" podStartSLOduration=1.71061526 podStartE2EDuration="3.888349384s" totalImagesPullingTime="2.177734124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:25:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:25:35.887023534 +0000 UTC m=+573.751927583" watchObservedRunningTime="2026-09-16 18:25:35.888349384 +0000 UTC m=+573.753253507"
Sep 16 18:25:35.909063 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:35.909030    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wwvq5"]
Sep 16 18:25:35.911007 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:35.910991    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wwvq5"
Sep 16 18:25:35.913434 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:35.913414    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gw25g\""
Sep 16 18:25:35.913682 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:35.913655    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 18:25:35.913808 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:35.913789    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 18:25:35.923171 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:35.923149    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wwvq5"]
Sep 16 18:25:35.985721 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:35.985649    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wg95\" (UniqueName: \"kubernetes.io/projected/a45b3d4c-0462-42a2-8b44-af37c5fe7ba0-kube-api-access-6wg95\") pod \"cert-manager-webhook-6486ffd997-wwvq5\" (UID: \"a45b3d4c-0462-42a2-8b44-af37c5fe7ba0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wwvq5"
Sep 16 18:25:35.985721 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:35.985718    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a45b3d4c-0462-42a2-8b44-af37c5fe7ba0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wwvq5\" (UID: \"a45b3d4c-0462-42a2-8b44-af37c5fe7ba0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wwvq5"
Sep 16 18:25:36.086256 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.086219    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a45b3d4c-0462-42a2-8b44-af37c5fe7ba0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wwvq5\" (UID: \"a45b3d4c-0462-42a2-8b44-af37c5fe7ba0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wwvq5"
Sep 16 18:25:36.086256 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.086259    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wg95\" (UniqueName: \"kubernetes.io/projected/a45b3d4c-0462-42a2-8b44-af37c5fe7ba0-kube-api-access-6wg95\") pod \"cert-manager-webhook-6486ffd997-wwvq5\" (UID: \"a45b3d4c-0462-42a2-8b44-af37c5fe7ba0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wwvq5"
Sep 16 18:25:36.107438 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.107411    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a45b3d4c-0462-42a2-8b44-af37c5fe7ba0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wwvq5\" (UID: \"a45b3d4c-0462-42a2-8b44-af37c5fe7ba0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wwvq5"
Sep 16 18:25:36.107737 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.107716    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wg95\" (UniqueName: \"kubernetes.io/projected/a45b3d4c-0462-42a2-8b44-af37c5fe7ba0-kube-api-access-6wg95\") pod \"cert-manager-webhook-6486ffd997-wwvq5\" (UID: \"a45b3d4c-0462-42a2-8b44-af37c5fe7ba0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wwvq5"
Sep 16 18:25:36.212537 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.212503    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:36.220534 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.220506    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wwvq5"
Sep 16 18:25:36.308813 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.308781    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:36.497380 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.497344    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wwvq5"]
Sep 16 18:25:36.500306 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:25:36.500276    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda45b3d4c_0462_42a2_8b44_af37c5fe7ba0.slice/crio-66d2be644d89629b83adf4a4c322e70b1bd3ad2588e066e3b78e4ce84df78420 WatchSource:0}: Error finding container 66d2be644d89629b83adf4a4c322e70b1bd3ad2588e066e3b78e4ce84df78420: Status 404 returned error can't find the container with id 66d2be644d89629b83adf4a4c322e70b1bd3ad2588e066e3b78e4ce84df78420
Sep 16 18:25:36.866315 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.866274    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wwvq5" event={"ID":"a45b3d4c-0462-42a2-8b44-af37c5fe7ba0","Type":"ContainerStarted","Data":"66d2be644d89629b83adf4a4c322e70b1bd3ad2588e066e3b78e4ce84df78420"}
Sep 16 18:25:36.881001 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.880965    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"]
Sep 16 18:25:36.884225 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.884192    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"
Sep 16 18:25:36.887237 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.887210    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:25:36.887366 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.887209    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:25:36.887366 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.887327    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-28c5h\""
Sep 16 18:25:36.891294 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.891273    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4086e04-688b-404b-803a-6689c7bf288a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75\" (UID: \"d4086e04-688b-404b-803a-6689c7bf288a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"
Sep 16 18:25:36.891365 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.891336    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzkvt\" (UniqueName: \"kubernetes.io/projected/d4086e04-688b-404b-803a-6689c7bf288a-kube-api-access-kzkvt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75\" (UID: \"d4086e04-688b-404b-803a-6689c7bf288a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"
Sep 16 18:25:36.891413 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.891380    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4086e04-688b-404b-803a-6689c7bf288a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75\" (UID: \"d4086e04-688b-404b-803a-6689c7bf288a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"
Sep 16 18:25:36.894999 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.894976    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"]
Sep 16 18:25:36.991784 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.991742    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzkvt\" (UniqueName: \"kubernetes.io/projected/d4086e04-688b-404b-803a-6689c7bf288a-kube-api-access-kzkvt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75\" (UID: \"d4086e04-688b-404b-803a-6689c7bf288a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"
Sep 16 18:25:36.991784 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.991789    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4086e04-688b-404b-803a-6689c7bf288a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75\" (UID: \"d4086e04-688b-404b-803a-6689c7bf288a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"
Sep 16 18:25:36.992035 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.991821    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4086e04-688b-404b-803a-6689c7bf288a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75\" (UID: \"d4086e04-688b-404b-803a-6689c7bf288a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"
Sep 16 18:25:36.992194 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.992172    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4086e04-688b-404b-803a-6689c7bf288a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75\" (UID: \"d4086e04-688b-404b-803a-6689c7bf288a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"
Sep 16 18:25:36.992248 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:36.992215    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4086e04-688b-404b-803a-6689c7bf288a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75\" (UID: \"d4086e04-688b-404b-803a-6689c7bf288a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"
Sep 16 18:25:37.001371 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:37.001337    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzkvt\" (UniqueName: \"kubernetes.io/projected/d4086e04-688b-404b-803a-6689c7bf288a-kube-api-access-kzkvt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75\" (UID: \"d4086e04-688b-404b-803a-6689c7bf288a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"
Sep 16 18:25:37.193813 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:37.193735    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"
Sep 16 18:25:37.474977 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:37.474934    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"]
Sep 16 18:25:37.478726 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:25:37.478694    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4086e04_688b_404b_803a_6689c7bf288a.slice/crio-5019d2fea45ead7a5107690b63992dbfa9f861468d9422e359d94ae5887c9753 WatchSource:0}: Error finding container 5019d2fea45ead7a5107690b63992dbfa9f861468d9422e359d94ae5887c9753: Status 404 returned error can't find the container with id 5019d2fea45ead7a5107690b63992dbfa9f861468d9422e359d94ae5887c9753
Sep 16 18:25:37.875148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:37.875056    2562 generic.go:356] "Generic (PLEG): container finished" podID="d4086e04-688b-404b-803a-6689c7bf288a" containerID="a99b6209e71f43ad5608a758630a5f205375a6bdc32e840ded7e366a9ca7f7e8" exitCode=0
Sep 16 18:25:37.875148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:37.875108    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75" event={"ID":"d4086e04-688b-404b-803a-6689c7bf288a","Type":"ContainerDied","Data":"a99b6209e71f43ad5608a758630a5f205375a6bdc32e840ded7e366a9ca7f7e8"}
Sep 16 18:25:37.875678 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:37.875150    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75" event={"ID":"d4086e04-688b-404b-803a-6689c7bf288a","Type":"ContainerStarted","Data":"5019d2fea45ead7a5107690b63992dbfa9f861468d9422e359d94ae5887c9753"}
Sep 16 18:25:37.935585 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:37.935535    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:38.151483 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:38.151400    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:39.887331 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:39.887296    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wwvq5" event={"ID":"a45b3d4c-0462-42a2-8b44-af37c5fe7ba0","Type":"ContainerStarted","Data":"80f89ee8f599458ed2bdc5b0d80a8e084e9a0bbd1dbf033a2d0ec02bf0affade"}
Sep 16 18:25:39.887801 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:39.887449    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wwvq5"
Sep 16 18:25:39.904427 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:39.904370    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-wwvq5" podStartSLOduration=2.264349422 podStartE2EDuration="4.904351894s" totalImagesPullingTime="2.640002472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:25:39.903216814 +0000 UTC m=+577.768120886" watchObservedRunningTime="2026-09-16 18:25:39.904351894 +0000 UTC m=+577.769255943"
Sep 16 18:25:40.841471 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:40.841384    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:40.895442 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:40.895411    2562 generic.go:356] "Generic (PLEG): container finished" podID="d4086e04-688b-404b-803a-6689c7bf288a" containerID="713ef2a3d4586376d9e65eda7b6791f6061892a888266311cf0e0ec7011b76b7" exitCode=0
Sep 16 18:25:40.895940 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:40.895509    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75" event={"ID":"d4086e04-688b-404b-803a-6689c7bf288a","Type":"ContainerDied","Data":"713ef2a3d4586376d9e65eda7b6791f6061892a888266311cf0e0ec7011b76b7"}
Sep 16 18:25:40.916088 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:40.916059    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:41.901863 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:41.901829    2562 generic.go:356] "Generic (PLEG): container finished" podID="d4086e04-688b-404b-803a-6689c7bf288a" containerID="ebef915a291dda2ecf3c00530d711d5c1c95d5f84038a25a402c99b6fdf52016" exitCode=0
Sep 16 18:25:41.902245 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:41.901923    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75" event={"ID":"d4086e04-688b-404b-803a-6689c7bf288a","Type":"ContainerDied","Data":"ebef915a291dda2ecf3c00530d711d5c1c95d5f84038a25a402c99b6fdf52016"}
Sep 16 18:25:43.022994 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:43.022966    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"
Sep 16 18:25:43.133155 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:43.133127    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4086e04-688b-404b-803a-6689c7bf288a-util\") pod \"d4086e04-688b-404b-803a-6689c7bf288a\" (UID: \"d4086e04-688b-404b-803a-6689c7bf288a\") "
Sep 16 18:25:43.133155 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:43.133159    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzkvt\" (UniqueName: \"kubernetes.io/projected/d4086e04-688b-404b-803a-6689c7bf288a-kube-api-access-kzkvt\") pod \"d4086e04-688b-404b-803a-6689c7bf288a\" (UID: \"d4086e04-688b-404b-803a-6689c7bf288a\") "
Sep 16 18:25:43.133343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:43.133178    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4086e04-688b-404b-803a-6689c7bf288a-bundle\") pod \"d4086e04-688b-404b-803a-6689c7bf288a\" (UID: \"d4086e04-688b-404b-803a-6689c7bf288a\") "
Sep 16 18:25:43.133624 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:43.133598    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4086e04-688b-404b-803a-6689c7bf288a-bundle" (OuterVolumeSpecName: "bundle") pod "d4086e04-688b-404b-803a-6689c7bf288a" (UID: "d4086e04-688b-404b-803a-6689c7bf288a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:25:43.135232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:43.135211    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4086e04-688b-404b-803a-6689c7bf288a-kube-api-access-kzkvt" (OuterVolumeSpecName: "kube-api-access-kzkvt") pod "d4086e04-688b-404b-803a-6689c7bf288a" (UID: "d4086e04-688b-404b-803a-6689c7bf288a"). InnerVolumeSpecName "kube-api-access-kzkvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:25:43.137472 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:43.137435    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4086e04-688b-404b-803a-6689c7bf288a-util" (OuterVolumeSpecName: "util") pod "d4086e04-688b-404b-803a-6689c7bf288a" (UID: "d4086e04-688b-404b-803a-6689c7bf288a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:25:43.234099 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:43.234070    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4086e04-688b-404b-803a-6689c7bf288a-util\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:25:43.234099 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:43.234097    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzkvt\" (UniqueName: \"kubernetes.io/projected/d4086e04-688b-404b-803a-6689c7bf288a-kube-api-access-kzkvt\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:25:43.234296 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:43.234108    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4086e04-688b-404b-803a-6689c7bf288a-bundle\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:25:43.290659 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:43.290626    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:43.407313 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:43.407280    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:43.911336 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:43.911310    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75"
Sep 16 18:25:43.911499 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:43.911309    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqbs75" event={"ID":"d4086e04-688b-404b-803a-6689c7bf288a","Type":"ContainerDied","Data":"5019d2fea45ead7a5107690b63992dbfa9f861468d9422e359d94ae5887c9753"}
Sep 16 18:25:43.911499 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:43.911424    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5019d2fea45ead7a5107690b63992dbfa9f861468d9422e359d94ae5887c9753"
Sep 16 18:25:45.898935 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:45.898907    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wwvq5"
Sep 16 18:25:45.944225 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:45.944192    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:47.205014 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.204981    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-l42bf"]
Sep 16 18:25:47.205375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.205243    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4086e04-688b-404b-803a-6689c7bf288a" containerName="pull"
Sep 16 18:25:47.205375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.205254    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4086e04-688b-404b-803a-6689c7bf288a" containerName="pull"
Sep 16 18:25:47.205375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.205263    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4086e04-688b-404b-803a-6689c7bf288a" containerName="extract"
Sep 16 18:25:47.205375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.205268    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4086e04-688b-404b-803a-6689c7bf288a" containerName="extract"
Sep 16 18:25:47.205375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.205287    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4086e04-688b-404b-803a-6689c7bf288a" containerName="util"
Sep 16 18:25:47.205375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.205293    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4086e04-688b-404b-803a-6689c7bf288a" containerName="util"
Sep 16 18:25:47.205375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.205338    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4086e04-688b-404b-803a-6689c7bf288a" containerName="extract"
Sep 16 18:25:47.209246 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.209229    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-l42bf"
Sep 16 18:25:47.213383 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.213360    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-fsl9j\""
Sep 16 18:25:47.228229 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.228204    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-l42bf"]
Sep 16 18:25:47.257519 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.257481    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p64gk\" (UniqueName: \"kubernetes.io/projected/e2593344-13f9-4a5c-bd94-203c52dd9147-kube-api-access-p64gk\") pod \"cert-manager-55b4d954d5-l42bf\" (UID: \"e2593344-13f9-4a5c-bd94-203c52dd9147\") " pod="cert-manager/cert-manager-55b4d954d5-l42bf"
Sep 16 18:25:47.257790 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.257534    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2593344-13f9-4a5c-bd94-203c52dd9147-bound-sa-token\") pod \"cert-manager-55b4d954d5-l42bf\" (UID: \"e2593344-13f9-4a5c-bd94-203c52dd9147\") " pod="cert-manager/cert-manager-55b4d954d5-l42bf"
Sep 16 18:25:47.358833 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.358797    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p64gk\" (UniqueName: \"kubernetes.io/projected/e2593344-13f9-4a5c-bd94-203c52dd9147-kube-api-access-p64gk\") pod \"cert-manager-55b4d954d5-l42bf\" (UID: \"e2593344-13f9-4a5c-bd94-203c52dd9147\") " pod="cert-manager/cert-manager-55b4d954d5-l42bf"
Sep 16 18:25:47.358953 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.358854    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2593344-13f9-4a5c-bd94-203c52dd9147-bound-sa-token\") pod \"cert-manager-55b4d954d5-l42bf\" (UID: \"e2593344-13f9-4a5c-bd94-203c52dd9147\") " pod="cert-manager/cert-manager-55b4d954d5-l42bf"
Sep 16 18:25:47.367624 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.367593    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2593344-13f9-4a5c-bd94-203c52dd9147-bound-sa-token\") pod \"cert-manager-55b4d954d5-l42bf\" (UID: \"e2593344-13f9-4a5c-bd94-203c52dd9147\") " pod="cert-manager/cert-manager-55b4d954d5-l42bf"
Sep 16 18:25:47.367738 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.367720    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p64gk\" (UniqueName: \"kubernetes.io/projected/e2593344-13f9-4a5c-bd94-203c52dd9147-kube-api-access-p64gk\") pod \"cert-manager-55b4d954d5-l42bf\" (UID: \"e2593344-13f9-4a5c-bd94-203c52dd9147\") " pod="cert-manager/cert-manager-55b4d954d5-l42bf"
Sep 16 18:25:47.518171 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.518071    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-l42bf"
Sep 16 18:25:47.787267 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.787241    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-l42bf"]
Sep 16 18:25:47.789755 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:25:47.789728    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2593344_13f9_4a5c_bd94_203c52dd9147.slice/crio-fdfcb7e70482c5badefc07e6d1a9b8878bab13846dfaa84d59b73deade4640f2 WatchSource:0}: Error finding container fdfcb7e70482c5badefc07e6d1a9b8878bab13846dfaa84d59b73deade4640f2: Status 404 returned error can't find the container with id fdfcb7e70482c5badefc07e6d1a9b8878bab13846dfaa84d59b73deade4640f2
Sep 16 18:25:47.930881 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.930841    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-l42bf" event={"ID":"e2593344-13f9-4a5c-bd94-203c52dd9147","Type":"ContainerStarted","Data":"c9d5ff175d43431f55ce3af1c7023a99a57eb78038a637e0a160800e9a513266"}
Sep 16 18:25:47.930881 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.930886    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-l42bf" event={"ID":"e2593344-13f9-4a5c-bd94-203c52dd9147","Type":"ContainerStarted","Data":"fdfcb7e70482c5badefc07e6d1a9b8878bab13846dfaa84d59b73deade4640f2"}
Sep 16 18:25:47.946479 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:47.946427    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-l42bf" podStartSLOduration=0.946409205 podStartE2EDuration="946.409205ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:25:47.945477208 +0000 UTC m=+585.810381283" watchObservedRunningTime="2026-09-16 18:25:47.946409205 +0000 UTC m=+585.811313255"
Sep 16 18:25:48.732111 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:48.732074    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:48.827876 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:48.827843    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:55.102226 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:55.102191    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:55.202854 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:55.202820    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:57.281350 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:25:57.281319    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:02.515764 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:02.515735    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:26:02.522665 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:02.522641    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:26:02.523789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:02.523768    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:26:02.530241 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:02.530219    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:26:07.500026 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.499984    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:07.544254 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.544219    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"]
Sep 16 18:26:07.546868 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.546847    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"
Sep 16 18:26:07.550215 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.550192    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:26:07.550459 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.550434    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:26:07.551316 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.551297    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-28c5h\""
Sep 16 18:26:07.558896 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.558872    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"]
Sep 16 18:26:07.576432 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.576394    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:07.612733 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.612696    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr\" (UID: \"2f1423c6-3ead-4ce5-87e8-a83a181bbd73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"
Sep 16 18:26:07.612733 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.612739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr\" (UID: \"2f1423c6-3ead-4ce5-87e8-a83a181bbd73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"
Sep 16 18:26:07.612942 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.612797    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4lc6\" (UniqueName: \"kubernetes.io/projected/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-kube-api-access-j4lc6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr\" (UID: \"2f1423c6-3ead-4ce5-87e8-a83a181bbd73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"
Sep 16 18:26:07.713471 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.713434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr\" (UID: \"2f1423c6-3ead-4ce5-87e8-a83a181bbd73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"
Sep 16 18:26:07.713471 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.713476    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr\" (UID: \"2f1423c6-3ead-4ce5-87e8-a83a181bbd73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"
Sep 16 18:26:07.713724 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.713503    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4lc6\" (UniqueName: \"kubernetes.io/projected/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-kube-api-access-j4lc6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr\" (UID: \"2f1423c6-3ead-4ce5-87e8-a83a181bbd73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"
Sep 16 18:26:07.713960 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.713935    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr\" (UID: \"2f1423c6-3ead-4ce5-87e8-a83a181bbd73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"
Sep 16 18:26:07.714001 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.713937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr\" (UID: \"2f1423c6-3ead-4ce5-87e8-a83a181bbd73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"
Sep 16 18:26:07.729027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.728996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4lc6\" (UniqueName: \"kubernetes.io/projected/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-kube-api-access-j4lc6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr\" (UID: \"2f1423c6-3ead-4ce5-87e8-a83a181bbd73\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"
Sep 16 18:26:07.855958 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:07.855873    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"
Sep 16 18:26:08.134516 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:08.134430    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"]
Sep 16 18:26:08.137310 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:26:08.137279    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1423c6_3ead_4ce5_87e8_a83a181bbd73.slice/crio-ab787029713521b828b22882304196a5053d3a7020f6999caa612446e4fc7e84 WatchSource:0}: Error finding container ab787029713521b828b22882304196a5053d3a7020f6999caa612446e4fc7e84: Status 404 returned error can't find the container with id ab787029713521b828b22882304196a5053d3a7020f6999caa612446e4fc7e84
Sep 16 18:26:09.018184 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:09.018106    2562 generic.go:356] "Generic (PLEG): container finished" podID="2f1423c6-3ead-4ce5-87e8-a83a181bbd73" containerID="7c42c5ec65a85d617b3346c8da2a17a65b75131f917a0711f62ecd8a19e20050" exitCode=0
Sep 16 18:26:09.018559 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:09.018190    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr" event={"ID":"2f1423c6-3ead-4ce5-87e8-a83a181bbd73","Type":"ContainerDied","Data":"7c42c5ec65a85d617b3346c8da2a17a65b75131f917a0711f62ecd8a19e20050"}
Sep 16 18:26:09.018559 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:09.018243    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr" event={"ID":"2f1423c6-3ead-4ce5-87e8-a83a181bbd73","Type":"ContainerStarted","Data":"ab787029713521b828b22882304196a5053d3a7020f6999caa612446e4fc7e84"}
Sep 16 18:26:09.964404 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:09.964352    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:10.029673 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:10.029635    2562 generic.go:356] "Generic (PLEG): container finished" podID="2f1423c6-3ead-4ce5-87e8-a83a181bbd73" containerID="d0bd2320b28746da240024aa24e7d6ae2e3568eb7eeb72f799a6e5c96f0556cb" exitCode=0
Sep 16 18:26:10.030143 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:10.029702    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr" event={"ID":"2f1423c6-3ead-4ce5-87e8-a83a181bbd73","Type":"ContainerDied","Data":"d0bd2320b28746da240024aa24e7d6ae2e3568eb7eeb72f799a6e5c96f0556cb"}
Sep 16 18:26:10.030143 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:10.029925    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:11.036674 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:11.036635    2562 generic.go:356] "Generic (PLEG): container finished" podID="2f1423c6-3ead-4ce5-87e8-a83a181bbd73" containerID="7a920e1eefd2ff84fe3de943d6df7b52182e94c95d76abd114997eaf3a2f8e9c" exitCode=0
Sep 16 18:26:11.037024 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:11.036728    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr" event={"ID":"2f1423c6-3ead-4ce5-87e8-a83a181bbd73","Type":"ContainerDied","Data":"7a920e1eefd2ff84fe3de943d6df7b52182e94c95d76abd114997eaf3a2f8e9c"}
Sep 16 18:26:12.161229 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:12.161204    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"
Sep 16 18:26:12.252235 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:12.252192    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-bundle\") pod \"2f1423c6-3ead-4ce5-87e8-a83a181bbd73\" (UID: \"2f1423c6-3ead-4ce5-87e8-a83a181bbd73\") "
Sep 16 18:26:12.252433 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:12.252249    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-util\") pod \"2f1423c6-3ead-4ce5-87e8-a83a181bbd73\" (UID: \"2f1423c6-3ead-4ce5-87e8-a83a181bbd73\") "
Sep 16 18:26:12.252433 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:12.252299    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4lc6\" (UniqueName: \"kubernetes.io/projected/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-kube-api-access-j4lc6\") pod \"2f1423c6-3ead-4ce5-87e8-a83a181bbd73\" (UID: \"2f1423c6-3ead-4ce5-87e8-a83a181bbd73\") "
Sep 16 18:26:12.252982 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:12.252950    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-bundle" (OuterVolumeSpecName: "bundle") pod "2f1423c6-3ead-4ce5-87e8-a83a181bbd73" (UID: "2f1423c6-3ead-4ce5-87e8-a83a181bbd73"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:26:12.254387 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:12.254366    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-kube-api-access-j4lc6" (OuterVolumeSpecName: "kube-api-access-j4lc6") pod "2f1423c6-3ead-4ce5-87e8-a83a181bbd73" (UID: "2f1423c6-3ead-4ce5-87e8-a83a181bbd73"). InnerVolumeSpecName "kube-api-access-j4lc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:26:12.257329 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:12.257303    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-util" (OuterVolumeSpecName: "util") pod "2f1423c6-3ead-4ce5-87e8-a83a181bbd73" (UID: "2f1423c6-3ead-4ce5-87e8-a83a181bbd73"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:26:12.353762 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:12.353662    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-util\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:26:12.353762 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:12.353697    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4lc6\" (UniqueName: \"kubernetes.io/projected/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-kube-api-access-j4lc6\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:26:12.353762 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:12.353714    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f1423c6-3ead-4ce5-87e8-a83a181bbd73-bundle\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:26:13.049696 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:13.048363    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr" event={"ID":"2f1423c6-3ead-4ce5-87e8-a83a181bbd73","Type":"ContainerDied","Data":"ab787029713521b828b22882304196a5053d3a7020f6999caa612446e4fc7e84"}
Sep 16 18:26:13.049696 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:13.048423    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab787029713521b828b22882304196a5053d3a7020f6999caa612446e4fc7e84"
Sep 16 18:26:13.049696 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:13.048501    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw67hr"
Sep 16 18:26:16.853828 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.853787    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:16.892123 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.892090    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"]
Sep 16 18:26:16.892380 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.892367    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1423c6-3ead-4ce5-87e8-a83a181bbd73" containerName="extract"
Sep 16 18:26:16.892419 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.892381    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1423c6-3ead-4ce5-87e8-a83a181bbd73" containerName="extract"
Sep 16 18:26:16.892419 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.892392    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1423c6-3ead-4ce5-87e8-a83a181bbd73" containerName="util"
Sep 16 18:26:16.892419 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.892397    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1423c6-3ead-4ce5-87e8-a83a181bbd73" containerName="util"
Sep 16 18:26:16.892419 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.892402    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1423c6-3ead-4ce5-87e8-a83a181bbd73" containerName="pull"
Sep 16 18:26:16.892419 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.892408    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1423c6-3ead-4ce5-87e8-a83a181bbd73" containerName="pull"
Sep 16 18:26:16.892582 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.892458    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f1423c6-3ead-4ce5-87e8-a83a181bbd73" containerName="extract"
Sep 16 18:26:16.895046 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.895029    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"
Sep 16 18:26:16.897663 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.897636    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:26:16.898604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.898586    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-28c5h\""
Sep 16 18:26:16.898703 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.898612    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:26:16.903802 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.903778    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"]
Sep 16 18:26:16.928040 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.928006    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:16.992706 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.992668    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhsq2\" (UniqueName: \"kubernetes.io/projected/a1ffca2f-d353-414f-8347-d664c9a18448-kube-api-access-bhsq2\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676\" (UID: \"a1ffca2f-d353-414f-8347-d664c9a18448\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"
Sep 16 18:26:16.992877 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.992727    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1ffca2f-d353-414f-8347-d664c9a18448-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676\" (UID: \"a1ffca2f-d353-414f-8347-d664c9a18448\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"
Sep 16 18:26:16.992877 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:16.992794    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1ffca2f-d353-414f-8347-d664c9a18448-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676\" (UID: \"a1ffca2f-d353-414f-8347-d664c9a18448\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"
Sep 16 18:26:17.093569 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:17.093499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1ffca2f-d353-414f-8347-d664c9a18448-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676\" (UID: \"a1ffca2f-d353-414f-8347-d664c9a18448\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"
Sep 16 18:26:17.093766 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:17.093589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1ffca2f-d353-414f-8347-d664c9a18448-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676\" (UID: \"a1ffca2f-d353-414f-8347-d664c9a18448\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"
Sep 16 18:26:17.093766 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:17.093646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhsq2\" (UniqueName: \"kubernetes.io/projected/a1ffca2f-d353-414f-8347-d664c9a18448-kube-api-access-bhsq2\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676\" (UID: \"a1ffca2f-d353-414f-8347-d664c9a18448\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"
Sep 16 18:26:17.093913 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:17.093891    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1ffca2f-d353-414f-8347-d664c9a18448-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676\" (UID: \"a1ffca2f-d353-414f-8347-d664c9a18448\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"
Sep 16 18:26:17.093967 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:17.093921    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1ffca2f-d353-414f-8347-d664c9a18448-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676\" (UID: \"a1ffca2f-d353-414f-8347-d664c9a18448\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"
Sep 16 18:26:17.104706 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:17.104636    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhsq2\" (UniqueName: \"kubernetes.io/projected/a1ffca2f-d353-414f-8347-d664c9a18448-kube-api-access-bhsq2\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676\" (UID: \"a1ffca2f-d353-414f-8347-d664c9a18448\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"
Sep 16 18:26:17.204693 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:17.204657    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"
Sep 16 18:26:17.477085 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:17.477060    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"]
Sep 16 18:26:17.479607 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:26:17.479580    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ffca2f_d353_414f_8347_d664c9a18448.slice/crio-b9b4d60608cba13ee424ca19e0925e44c843a29e55733a5ce106f843c158db29 WatchSource:0}: Error finding container b9b4d60608cba13ee424ca19e0925e44c843a29e55733a5ce106f843c158db29: Status 404 returned error can't find the container with id b9b4d60608cba13ee424ca19e0925e44c843a29e55733a5ce106f843c158db29
Sep 16 18:26:17.967843 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:17.967815    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:18.068382 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:18.068353    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:18.078366 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:18.078333    2562 generic.go:356] "Generic (PLEG): container finished" podID="a1ffca2f-d353-414f-8347-d664c9a18448" containerID="3dfe71c8261ac41daaca1c18bf44ef9d4d448f5af97ec5afc836ea1d799a9fd2" exitCode=0
Sep 16 18:26:18.078508 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:18.078420    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676" event={"ID":"a1ffca2f-d353-414f-8347-d664c9a18448","Type":"ContainerDied","Data":"3dfe71c8261ac41daaca1c18bf44ef9d4d448f5af97ec5afc836ea1d799a9fd2"}
Sep 16 18:26:18.078508 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:18.078469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676" event={"ID":"a1ffca2f-d353-414f-8347-d664c9a18448","Type":"ContainerStarted","Data":"b9b4d60608cba13ee424ca19e0925e44c843a29e55733a5ce106f843c158db29"}
Sep 16 18:26:18.605556 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:18.605515    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:18.881073 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:18.880992    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:18.981844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:18.981817    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:18.997565 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:18.997510    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"]
Sep 16 18:26:18.999884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:18.999864    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.002666 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.002635    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 18:26:19.002779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.002648    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 18:26:19.002916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.002896    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-r85q5\""
Sep 16 18:26:19.003037 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.002932    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 18:26:19.003106 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.003036    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 18:26:19.016561 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.016523    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"]
Sep 16 18:26:19.085316 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.085283    2562 generic.go:356] "Generic (PLEG): container finished" podID="a1ffca2f-d353-414f-8347-d664c9a18448" containerID="029750a95d736ad98f516657b5e8451b4a1d4ef70b71f1e172414206e6bb0535" exitCode=0
Sep 16 18:26:19.085473 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.085366    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676" event={"ID":"a1ffca2f-d353-414f-8347-d664c9a18448","Type":"ContainerDied","Data":"029750a95d736ad98f516657b5e8451b4a1d4ef70b71f1e172414206e6bb0535"}
Sep 16 18:26:19.112352 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.112321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9d750944-d535-40d7-b496-1088d364f6fa-manifests\") pod \"opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz\" (UID: \"9d750944-d535-40d7-b496-1088d364f6fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.112572 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.112524    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d750944-d535-40d7-b496-1088d364f6fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz\" (UID: \"9d750944-d535-40d7-b496-1088d364f6fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.112773 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.112711    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9d750944-d535-40d7-b496-1088d364f6fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz\" (UID: \"9d750944-d535-40d7-b496-1088d364f6fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.112871 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.112841    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqjgp\" (UniqueName: \"kubernetes.io/projected/9d750944-d535-40d7-b496-1088d364f6fa-kube-api-access-fqjgp\") pod \"opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz\" (UID: \"9d750944-d535-40d7-b496-1088d364f6fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.113286 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.113085    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d750944-d535-40d7-b496-1088d364f6fa-tmp\") pod \"opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz\" (UID: \"9d750944-d535-40d7-b496-1088d364f6fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.127473 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.127420    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:19.214623 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.214588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d750944-d535-40d7-b496-1088d364f6fa-tmp\") pod \"opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz\" (UID: \"9d750944-d535-40d7-b496-1088d364f6fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.214811 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.214635    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9d750944-d535-40d7-b496-1088d364f6fa-manifests\") pod \"opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz\" (UID: \"9d750944-d535-40d7-b496-1088d364f6fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.214811 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.214658    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d750944-d535-40d7-b496-1088d364f6fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz\" (UID: \"9d750944-d535-40d7-b496-1088d364f6fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.214811 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.214677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9d750944-d535-40d7-b496-1088d364f6fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz\" (UID: \"9d750944-d535-40d7-b496-1088d364f6fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.214811 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.214707    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqjgp\" (UniqueName: \"kubernetes.io/projected/9d750944-d535-40d7-b496-1088d364f6fa-kube-api-access-fqjgp\") pod \"opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz\" (UID: \"9d750944-d535-40d7-b496-1088d364f6fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.215324 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.215292    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9d750944-d535-40d7-b496-1088d364f6fa-manifests\") pod \"opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz\" (UID: \"9d750944-d535-40d7-b496-1088d364f6fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.217446 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.217405    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d750944-d535-40d7-b496-1088d364f6fa-tmp\") pod \"opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz\" (UID: \"9d750944-d535-40d7-b496-1088d364f6fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.217723 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.217694    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d750944-d535-40d7-b496-1088d364f6fa-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz\" (UID: \"9d750944-d535-40d7-b496-1088d364f6fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.217797 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.217711    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9d750944-d535-40d7-b496-1088d364f6fa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz\" (UID: \"9d750944-d535-40d7-b496-1088d364f6fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.222699 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.222676    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqjgp\" (UniqueName: \"kubernetes.io/projected/9d750944-d535-40d7-b496-1088d364f6fa-kube-api-access-fqjgp\") pod \"opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz\" (UID: \"9d750944-d535-40d7-b496-1088d364f6fa\") " pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.227802 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.227778    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:19.309991 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.309943    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:19.328093 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.328017    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:19.431699 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.431620    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:19.531906 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.531876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:19.588710 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.588674    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"]
Sep 16 18:26:19.591762 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:26:19.591735    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d750944_d535_40d7_b496_1088d364f6fa.slice/crio-a83fd3e48d6034218a896dfcc98d397b3d7f86402672e772a039c68ba358a813 WatchSource:0}: Error finding container a83fd3e48d6034218a896dfcc98d397b3d7f86402672e772a039c68ba358a813: Status 404 returned error can't find the container with id a83fd3e48d6034218a896dfcc98d397b3d7f86402672e772a039c68ba358a813
Sep 16 18:26:19.606761 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:19.606734    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:20.093325 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:20.093280    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz" event={"ID":"9d750944-d535-40d7-b496-1088d364f6fa","Type":"ContainerStarted","Data":"a83fd3e48d6034218a896dfcc98d397b3d7f86402672e772a039c68ba358a813"}
Sep 16 18:26:20.095894 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:20.095866    2562 generic.go:356] "Generic (PLEG): container finished" podID="a1ffca2f-d353-414f-8347-d664c9a18448" containerID="969b6dd6d5c2c74862e02397199346003e24e8edcbf9724527305fc9fb2827b9" exitCode=0
Sep 16 18:26:20.096034 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:20.095928    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676" event={"ID":"a1ffca2f-d353-414f-8347-d664c9a18448","Type":"ContainerDied","Data":"969b6dd6d5c2c74862e02397199346003e24e8edcbf9724527305fc9fb2827b9"}
Sep 16 18:26:22.444209 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:22.444180    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"
Sep 16 18:26:22.541704 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:22.541673    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1ffca2f-d353-414f-8347-d664c9a18448-util\") pod \"a1ffca2f-d353-414f-8347-d664c9a18448\" (UID: \"a1ffca2f-d353-414f-8347-d664c9a18448\") "
Sep 16 18:26:22.541853 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:22.541716    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhsq2\" (UniqueName: \"kubernetes.io/projected/a1ffca2f-d353-414f-8347-d664c9a18448-kube-api-access-bhsq2\") pod \"a1ffca2f-d353-414f-8347-d664c9a18448\" (UID: \"a1ffca2f-d353-414f-8347-d664c9a18448\") "
Sep 16 18:26:22.541853 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:22.541741    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1ffca2f-d353-414f-8347-d664c9a18448-bundle\") pod \"a1ffca2f-d353-414f-8347-d664c9a18448\" (UID: \"a1ffca2f-d353-414f-8347-d664c9a18448\") "
Sep 16 18:26:22.542533 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:22.542504    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ffca2f-d353-414f-8347-d664c9a18448-bundle" (OuterVolumeSpecName: "bundle") pod "a1ffca2f-d353-414f-8347-d664c9a18448" (UID: "a1ffca2f-d353-414f-8347-d664c9a18448"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:26:22.543921 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:22.543890    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ffca2f-d353-414f-8347-d664c9a18448-kube-api-access-bhsq2" (OuterVolumeSpecName: "kube-api-access-bhsq2") pod "a1ffca2f-d353-414f-8347-d664c9a18448" (UID: "a1ffca2f-d353-414f-8347-d664c9a18448"). InnerVolumeSpecName "kube-api-access-bhsq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:26:22.547093 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:22.547069    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ffca2f-d353-414f-8347-d664c9a18448-util" (OuterVolumeSpecName: "util") pod "a1ffca2f-d353-414f-8347-d664c9a18448" (UID: "a1ffca2f-d353-414f-8347-d664c9a18448"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:26:22.643119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:22.643079    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1ffca2f-d353-414f-8347-d664c9a18448-util\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:26:22.643119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:22.643112    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhsq2\" (UniqueName: \"kubernetes.io/projected/a1ffca2f-d353-414f-8347-d664c9a18448-kube-api-access-bhsq2\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:26:22.643119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:22.643127    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1ffca2f-d353-414f-8347-d664c9a18448-bundle\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:26:23.113527 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:23.113486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz" event={"ID":"9d750944-d535-40d7-b496-1088d364f6fa","Type":"ContainerStarted","Data":"a4c85c7a295187875ca6cc74da4e836a24da070194f417c6507c813e36b28bf5"}
Sep 16 18:26:23.115384 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:23.115361    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676"
Sep 16 18:26:23.115499 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:23.115360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07j676" event={"ID":"a1ffca2f-d353-414f-8347-d664c9a18448","Type":"ContainerDied","Data":"b9b4d60608cba13ee424ca19e0925e44c843a29e55733a5ce106f843c158db29"}
Sep 16 18:26:23.115499 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:23.115472    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9b4d60608cba13ee424ca19e0925e44c843a29e55733a5ce106f843c158db29"
Sep 16 18:26:24.121682 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:24.121647    2562 generic.go:356] "Generic (PLEG): container finished" podID="9d750944-d535-40d7-b496-1088d364f6fa" containerID="a4c85c7a295187875ca6cc74da4e836a24da070194f417c6507c813e36b28bf5" exitCode=0
Sep 16 18:26:24.122070 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:24.121739    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz" event={"ID":"9d750944-d535-40d7-b496-1088d364f6fa","Type":"ContainerDied","Data":"a4c85c7a295187875ca6cc74da4e836a24da070194f417c6507c813e36b28bf5"}
Sep 16 18:26:25.127899 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:25.127861    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz" event={"ID":"9d750944-d535-40d7-b496-1088d364f6fa","Type":"ContainerStarted","Data":"45df93eac9d690de5f5c55e0a727a3070fdc3a106c399e27c1dd191b22942bfb"}
Sep 16 18:26:25.128372 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:25.127991    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:25.153216 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:25.153132    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz" podStartSLOduration=2.746791952 podStartE2EDuration="7.153116231s" totalImagesPullingTime="3.371958214s" totalInitContainerRuntime="1.034366065s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:26:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:26:25.150785542 +0000 UTC m=+623.015689592" watchObservedRunningTime="2026-09-16 18:26:25.153116231 +0000 UTC m=+623.018020336"
Sep 16 18:26:36.135759 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:36.135726    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz"
Sep 16 18:26:38.732585 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:38.732529    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:38.896395 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:38.896359    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:39.378106 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.378069    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:39.531583 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.531535    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k"]
Sep 16 18:26:39.531834 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.531823    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1ffca2f-d353-414f-8347-d664c9a18448" containerName="pull"
Sep 16 18:26:39.531878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.531836    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1ffca2f-d353-414f-8347-d664c9a18448" containerName="pull"
Sep 16 18:26:39.531878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.531857    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1ffca2f-d353-414f-8347-d664c9a18448" containerName="util"
Sep 16 18:26:39.531878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.531862    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1ffca2f-d353-414f-8347-d664c9a18448" containerName="util"
Sep 16 18:26:39.531878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.531869    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1ffca2f-d353-414f-8347-d664c9a18448" containerName="extract"
Sep 16 18:26:39.531878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.531875    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1ffca2f-d353-414f-8347-d664c9a18448" containerName="extract"
Sep 16 18:26:39.532028 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.531931    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1ffca2f-d353-414f-8347-d664c9a18448" containerName="extract"
Sep 16 18:26:39.541400 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.541377    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k"
Sep 16 18:26:39.543825 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.543793    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 18:26:39.543938 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.543909    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 18:26:39.551684 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.551663    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k"]
Sep 16 18:26:39.657243 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.657168    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:39.687313 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.687272    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2192ff6e-623b-4bd3-8918-4e584915d514-tmp\") pod \"kube-auth-proxy-79f4c98848-bqq9k\" (UID: \"2192ff6e-623b-4bd3-8918-4e584915d514\") " pod="openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k"
Sep 16 18:26:39.687313 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.687317    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhv6w\" (UniqueName: \"kubernetes.io/projected/2192ff6e-623b-4bd3-8918-4e584915d514-kube-api-access-bhv6w\") pod \"kube-auth-proxy-79f4c98848-bqq9k\" (UID: \"2192ff6e-623b-4bd3-8918-4e584915d514\") " pod="openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k"
Sep 16 18:26:39.687517 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.687356    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2192ff6e-623b-4bd3-8918-4e584915d514-tls-certs\") pod \"kube-auth-proxy-79f4c98848-bqq9k\" (UID: \"2192ff6e-623b-4bd3-8918-4e584915d514\") " pod="openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k"
Sep 16 18:26:39.788624 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.788595    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2192ff6e-623b-4bd3-8918-4e584915d514-tmp\") pod \"kube-auth-proxy-79f4c98848-bqq9k\" (UID: \"2192ff6e-623b-4bd3-8918-4e584915d514\") " pod="openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k"
Sep 16 18:26:39.789065 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.788637    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhv6w\" (UniqueName: \"kubernetes.io/projected/2192ff6e-623b-4bd3-8918-4e584915d514-kube-api-access-bhv6w\") pod \"kube-auth-proxy-79f4c98848-bqq9k\" (UID: \"2192ff6e-623b-4bd3-8918-4e584915d514\") " pod="openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k"
Sep 16 18:26:39.789065 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.788681    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2192ff6e-623b-4bd3-8918-4e584915d514-tls-certs\") pod \"kube-auth-proxy-79f4c98848-bqq9k\" (UID: \"2192ff6e-623b-4bd3-8918-4e584915d514\") " pod="openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k"
Sep 16 18:26:39.790710 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.790687    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2192ff6e-623b-4bd3-8918-4e584915d514-tmp\") pod \"kube-auth-proxy-79f4c98848-bqq9k\" (UID: \"2192ff6e-623b-4bd3-8918-4e584915d514\") " pod="openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k"
Sep 16 18:26:39.790953 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.790934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2192ff6e-623b-4bd3-8918-4e584915d514-tls-certs\") pod \"kube-auth-proxy-79f4c98848-bqq9k\" (UID: \"2192ff6e-623b-4bd3-8918-4e584915d514\") " pod="openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k"
Sep 16 18:26:39.807329 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.807302    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhv6w\" (UniqueName: \"kubernetes.io/projected/2192ff6e-623b-4bd3-8918-4e584915d514-kube-api-access-bhv6w\") pod \"kube-auth-proxy-79f4c98848-bqq9k\" (UID: \"2192ff6e-623b-4bd3-8918-4e584915d514\") " pod="openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k"
Sep 16 18:26:39.851295 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:39.851264    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k"
Sep 16 18:26:40.127425 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:40.127244    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k"]
Sep 16 18:26:40.130114 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:26:40.130088    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2192ff6e_623b_4bd3_8918_4e584915d514.slice/crio-c442624c6927b3c29c674d617486632019ee49770c6d16d21c660a48027426c7 WatchSource:0}: Error finding container c442624c6927b3c29c674d617486632019ee49770c6d16d21c660a48027426c7: Status 404 returned error can't find the container with id c442624c6927b3c29c674d617486632019ee49770c6d16d21c660a48027426c7
Sep 16 18:26:40.198136 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:40.198098    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k" event={"ID":"2192ff6e-623b-4bd3-8918-4e584915d514","Type":"ContainerStarted","Data":"c442624c6927b3c29c674d617486632019ee49770c6d16d21c660a48027426c7"}
Sep 16 18:26:40.944130 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:40.944089    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:40.978848 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:40.978802    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"]
Sep 16 18:26:40.981440 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:40.981415    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:40.985779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:40.985241    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 18:26:40.985779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:40.985445    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 18:26:40.985779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:40.985486    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:26:40.985779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:40.985445    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 18:26:40.986118 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:40.985891    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-x44sq\""
Sep 16 18:26:41.078333 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.078296    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:41.100507 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.100422    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd689351-fd99-4084-a584-dd4fac79625f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.100507 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.100475    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd689351-fd99-4084-a584-dd4fac79625f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.100765 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.100580    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.100765 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.100616    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xwgt\" (UniqueName: \"kubernetes.io/projected/cd689351-fd99-4084-a584-dd4fac79625f-kube-api-access-7xwgt\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.100765 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.100648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd689351-fd99-4084-a584-dd4fac79625f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.100765 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.100673    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.100765 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.100729    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.120353 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.120279    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"]
Sep 16 18:26:41.201502 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.201467    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd689351-fd99-4084-a584-dd4fac79625f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.201718 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.201524    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd689351-fd99-4084-a584-dd4fac79625f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.201718 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.201591    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.201718 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.201616    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xwgt\" (UniqueName: \"kubernetes.io/projected/cd689351-fd99-4084-a584-dd4fac79625f-kube-api-access-7xwgt\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.201718 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.201645    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd689351-fd99-4084-a584-dd4fac79625f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.201718 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.201669    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.201718 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.201718    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.202559 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.202519    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd689351-fd99-4084-a584-dd4fac79625f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.204871 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.204800    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd689351-fd99-4084-a584-dd4fac79625f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.205423 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.205380    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.206370 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.206313    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.207822 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.207798    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.214766 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.214740    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd689351-fd99-4084-a584-dd4fac79625f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.215171 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.215052    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xwgt\" (UniqueName: \"kubernetes.io/projected/cd689351-fd99-4084-a584-dd4fac79625f-kube-api-access-7xwgt\") pod \"istiod-openshift-gateway-7b7995d5cd-s5vnk\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.294257 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.294217    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:41.601184 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:41.601155    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"]
Sep 16 18:26:42.298255 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:42.298217    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:42.761922 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:26:42.761892    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd689351_fd99_4084_a584_dd4fac79625f.slice/crio-6a070b803e38c7ac6be970075b198a3cf22b1082e4386338f6123a8449c922aa WatchSource:0}: Error finding container 6a070b803e38c7ac6be970075b198a3cf22b1082e4386338f6123a8449c922aa: Status 404 returned error can't find the container with id 6a070b803e38c7ac6be970075b198a3cf22b1082e4386338f6123a8449c922aa
Sep 16 18:26:43.218074 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:43.218035    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk" event={"ID":"cd689351-fd99-4084-a584-dd4fac79625f","Type":"ContainerStarted","Data":"6a070b803e38c7ac6be970075b198a3cf22b1082e4386338f6123a8449c922aa"}
Sep 16 18:26:43.219496 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:43.219475    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k" event={"ID":"2192ff6e-623b-4bd3-8918-4e584915d514","Type":"ContainerStarted","Data":"051820206ca784bf75e95e1e9541faab13595ca373a2edd2db3e732dce25cff8"}
Sep 16 18:26:43.279056 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:43.278991    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79f4c98848-bqq9k" podStartSLOduration=1.595676247 podStartE2EDuration="4.278976121s" totalImagesPullingTime="2.683299874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:26:43.275347744 +0000 UTC m=+641.140251793" watchObservedRunningTime="2026-09-16 18:26:43.278976121 +0000 UTC m=+641.143880168"
Sep 16 18:26:45.712611 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:45.711849    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:26:45.712611 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:45.711930    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:26:46.095391 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.095295    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:46.178855 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.178816    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:46.238184 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.238145    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk" event={"ID":"cd689351-fd99-4084-a584-dd4fac79625f","Type":"ContainerStarted","Data":"a4929848b868b31997842c0012c07d0112f51e3c1474b8aadbcd792959c1a7c8"}
Sep 16 18:26:46.238370 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.238205    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:46.239755 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.239728    2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-s5vnk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 18:26:46.239856 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.239781    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk" podUID="cd689351-fd99-4084-a584-dd4fac79625f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:26:46.331089 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.331029    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk" podStartSLOduration=3.3833859410000002 podStartE2EDuration="6.331006799s" totalImagesPullingTime="2.947620858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:26:46.260515412 +0000 UTC m=+644.125419461" watchObservedRunningTime="2026-09-16 18:26:46.331006799 +0000 UTC m=+644.195910847"
Sep 16 18:26:46.331525 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.331509    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"]
Sep 16 18:26:46.334951 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.334925    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.337797 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.337770    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hpwk5\""
Sep 16 18:26:46.360921 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.360848    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:46.383464 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.383437    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"]
Sep 16 18:26:46.447908 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.447869    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.448076 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.447953    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.448076 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.447995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl4vh\" (UniqueName: \"kubernetes.io/projected/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-kube-api-access-sl4vh\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.448076 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.448019    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.448190 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.448080    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.448190 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.448104    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.448190 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.448129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.448190 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.448162    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.448310 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.448197    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.477939 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.477874    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:46.548837 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.548800    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.549036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.548868    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.549036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.548923    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.549036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.548953    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.549036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.548988    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl4vh\" (UniqueName: \"kubernetes.io/projected/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-kube-api-access-sl4vh\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.549036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.549021    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.549247 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.549070    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.549247 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.549107    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.549247 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.549136    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.549247 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.549206    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.549812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.549773    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.549812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.549799    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.549981 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.549837    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.549981 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.549886    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.551802 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.551781    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.551975 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.551957    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.557214 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.557190    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.557321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.557302    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl4vh\" (UniqueName: \"kubernetes.io/projected/bc9125d2-cfd5-428d-a511-0f7943dc8ff7-kube-api-access-sl4vh\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799crdmz4\" (UID: \"bc9125d2-cfd5-428d-a511-0f7943dc8ff7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.578969 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.578932    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:46.645659 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.645559    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:46.919487 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:46.919462    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"]
Sep 16 18:26:46.921771 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:26:46.921740    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9125d2_cfd5_428d_a511_0f7943dc8ff7.slice/crio-6f0b9bd470ac1a368177ce9591936bec0c31329f0ee76284a200abf2922e6a5b WatchSource:0}: Error finding container 6f0b9bd470ac1a368177ce9591936bec0c31329f0ee76284a200abf2922e6a5b: Status 404 returned error can't find the container with id 6f0b9bd470ac1a368177ce9591936bec0c31329f0ee76284a200abf2922e6a5b
Sep 16 18:26:47.244001 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:47.243966    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4" event={"ID":"bc9125d2-cfd5-428d-a511-0f7943dc8ff7","Type":"ContainerStarted","Data":"6f0b9bd470ac1a368177ce9591936bec0c31329f0ee76284a200abf2922e6a5b"}
Sep 16 18:26:47.244749 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:47.244731    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:26:49.824862 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:49.824827    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:49.919754 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:49.919718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:50.020903 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:50.020843    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:50.127130 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:50.127022    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:50.201772 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:50.201737    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:26:50.201882 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:50.201809    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:26:50.201882 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:50.201836    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:26:50.362030 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:50.361981    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:50.797130 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:50.797097    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:51.268299 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:51.268265    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4" event={"ID":"bc9125d2-cfd5-428d-a511-0f7943dc8ff7","Type":"ContainerStarted","Data":"156b493ac3f74370621a165e482d1acb094f26c5fe0fe6a3e2110468716c2450"}
Sep 16 18:26:51.292818 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:51.292766    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4" podStartSLOduration=2.01512425 podStartE2EDuration="5.292750229s" totalImagesPullingTime="3.277625979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:26:51.291272362 +0000 UTC m=+649.156176411" watchObservedRunningTime="2026-09-16 18:26:51.292750229 +0000 UTC m=+649.157654277"
Sep 16 18:26:51.646428 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:51.646331    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:51.650836 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:51.650806    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:52.273440 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:52.273405    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:26:52.274348 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:26:52.274331    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799crdmz4"
Sep 16 18:27:00.113931 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.113891    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:00.183865 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.183832    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk"]
Sep 16 18:27:00.186944 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.186926    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk"
Sep 16 18:27:00.190052 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.190021    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 16 18:27:00.190052 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.190035    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dpzg7\""
Sep 16 18:27:00.198196 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.198167    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:00.200999 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.200974    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk"]
Sep 16 18:27:00.202649 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.202625    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9"]
Sep 16 18:27:00.205769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.205752    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9"
Sep 16 18:27:00.208581 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.208538    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 16 18:27:00.208679 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.208633    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7pwrt\""
Sep 16 18:27:00.223727 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.223699    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9"]
Sep 16 18:27:00.252871 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.252836    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51297c36-21d1-48af-8f30-df5470ab7218-cert\") pod \"kserve-controller-manager-d6f4b65b8-mmvjk\" (UID: \"51297c36-21d1-48af-8f30-df5470ab7218\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk"
Sep 16 18:27:00.252871 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.252871    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f0b145a-64ef-4406-ad4b-23d6357b5ec2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-r2lj9\" (UID: \"0f0b145a-64ef-4406-ad4b-23d6357b5ec2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9"
Sep 16 18:27:00.253072 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.252909    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbbr4\" (UniqueName: \"kubernetes.io/projected/51297c36-21d1-48af-8f30-df5470ab7218-kube-api-access-gbbr4\") pod \"kserve-controller-manager-d6f4b65b8-mmvjk\" (UID: \"51297c36-21d1-48af-8f30-df5470ab7218\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk"
Sep 16 18:27:00.253072 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.252985    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtgg\" (UniqueName: \"kubernetes.io/projected/0f0b145a-64ef-4406-ad4b-23d6357b5ec2-kube-api-access-nrtgg\") pod \"llmisvc-controller-manager-777c6bd7d5-r2lj9\" (UID: \"0f0b145a-64ef-4406-ad4b-23d6357b5ec2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9"
Sep 16 18:27:00.353722 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.353686    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtgg\" (UniqueName: \"kubernetes.io/projected/0f0b145a-64ef-4406-ad4b-23d6357b5ec2-kube-api-access-nrtgg\") pod \"llmisvc-controller-manager-777c6bd7d5-r2lj9\" (UID: \"0f0b145a-64ef-4406-ad4b-23d6357b5ec2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9"
Sep 16 18:27:00.353915 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.353747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51297c36-21d1-48af-8f30-df5470ab7218-cert\") pod \"kserve-controller-manager-d6f4b65b8-mmvjk\" (UID: \"51297c36-21d1-48af-8f30-df5470ab7218\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk"
Sep 16 18:27:00.353915 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.353764    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f0b145a-64ef-4406-ad4b-23d6357b5ec2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-r2lj9\" (UID: \"0f0b145a-64ef-4406-ad4b-23d6357b5ec2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9"
Sep 16 18:27:00.353915 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.353790    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbbr4\" (UniqueName: \"kubernetes.io/projected/51297c36-21d1-48af-8f30-df5470ab7218-kube-api-access-gbbr4\") pod \"kserve-controller-manager-d6f4b65b8-mmvjk\" (UID: \"51297c36-21d1-48af-8f30-df5470ab7218\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk"
Sep 16 18:27:00.354090 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:27:00.353917    2562 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 16 18:27:00.354090 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:27:00.353933    2562 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 18:27:00.354090 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:27:00.353992    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51297c36-21d1-48af-8f30-df5470ab7218-cert podName:51297c36-21d1-48af-8f30-df5470ab7218 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:00.85397097 +0000 UTC m=+658.718875015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/51297c36-21d1-48af-8f30-df5470ab7218-cert") pod "kserve-controller-manager-d6f4b65b8-mmvjk" (UID: "51297c36-21d1-48af-8f30-df5470ab7218") : secret "kserve-webhook-server-cert" not found
Sep 16 18:27:00.354090 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:27:00.354055    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f0b145a-64ef-4406-ad4b-23d6357b5ec2-cert podName:0f0b145a-64ef-4406-ad4b-23d6357b5ec2 nodeName:}" failed. No retries permitted until 2026-09-16 18:27:00.854000519 +0000 UTC m=+658.718904560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0f0b145a-64ef-4406-ad4b-23d6357b5ec2-cert") pod "llmisvc-controller-manager-777c6bd7d5-r2lj9" (UID: "0f0b145a-64ef-4406-ad4b-23d6357b5ec2") : secret "llmisvc-webhook-server-cert" not found
Sep 16 18:27:00.371746 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.371669    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbbr4\" (UniqueName: \"kubernetes.io/projected/51297c36-21d1-48af-8f30-df5470ab7218-kube-api-access-gbbr4\") pod \"kserve-controller-manager-d6f4b65b8-mmvjk\" (UID: \"51297c36-21d1-48af-8f30-df5470ab7218\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk"
Sep 16 18:27:00.371746 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.371669    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtgg\" (UniqueName: \"kubernetes.io/projected/0f0b145a-64ef-4406-ad4b-23d6357b5ec2-kube-api-access-nrtgg\") pod \"llmisvc-controller-manager-777c6bd7d5-r2lj9\" (UID: \"0f0b145a-64ef-4406-ad4b-23d6357b5ec2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9"
Sep 16 18:27:00.494366 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.494331    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:00.581717 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.581680    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:00.856855 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.856815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51297c36-21d1-48af-8f30-df5470ab7218-cert\") pod \"kserve-controller-manager-d6f4b65b8-mmvjk\" (UID: \"51297c36-21d1-48af-8f30-df5470ab7218\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk"
Sep 16 18:27:00.856855 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.856857    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f0b145a-64ef-4406-ad4b-23d6357b5ec2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-r2lj9\" (UID: \"0f0b145a-64ef-4406-ad4b-23d6357b5ec2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9"
Sep 16 18:27:00.859303 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.859280    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0f0b145a-64ef-4406-ad4b-23d6357b5ec2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-r2lj9\" (UID: \"0f0b145a-64ef-4406-ad4b-23d6357b5ec2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9"
Sep 16 18:27:00.859390 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:00.859280    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51297c36-21d1-48af-8f30-df5470ab7218-cert\") pod \"kserve-controller-manager-d6f4b65b8-mmvjk\" (UID: \"51297c36-21d1-48af-8f30-df5470ab7218\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk"
Sep 16 18:27:01.097265 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.097227    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk"
Sep 16 18:27:01.116272 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.116242    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9"
Sep 16 18:27:01.341977 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.341944    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"]
Sep 16 18:27:01.342174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.342152    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk" podUID="cd689351-fd99-4084-a584-dd4fac79625f" containerName="discovery" containerID="cri-o://a4929848b868b31997842c0012c07d0112f51e3c1474b8aadbcd792959c1a7c8" gracePeriod=30
Sep 16 18:27:01.382139 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.382107    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk"]
Sep 16 18:27:01.410602 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.410577    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9"]
Sep 16 18:27:01.418632 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.418606    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:01.440660 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:27:01.440627    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51297c36_21d1_48af_8f30_df5470ab7218.slice/crio-0d13596d44562ebb5eb19da7d88b9f98f39e7c55ac969dbf876424c3261ebf51 WatchSource:0}: Error finding container 0d13596d44562ebb5eb19da7d88b9f98f39e7c55ac969dbf876424c3261ebf51: Status 404 returned error can't find the container with id 0d13596d44562ebb5eb19da7d88b9f98f39e7c55ac969dbf876424c3261ebf51
Sep 16 18:27:01.440920 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:27:01.440894    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0f0b145a_64ef_4406_ad4b_23d6357b5ec2.slice/crio-a99b7d7ce4b934acc41089ef79d600fac1924fb2eacc4bd85f6b5b2bc9a1fe48 WatchSource:0}: Error finding container a99b7d7ce4b934acc41089ef79d600fac1924fb2eacc4bd85f6b5b2bc9a1fe48: Status 404 returned error can't find the container with id a99b7d7ce4b934acc41089ef79d600fac1924fb2eacc4bd85f6b5b2bc9a1fe48
Sep 16 18:27:01.516378 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.516351    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:01.594737 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.594706    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:27:01.624113 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.624024    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:01.663429 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.663397    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-istio-csr-dns-cert\") pod \"cd689351-fd99-4084-a584-dd4fac79625f\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") "
Sep 16 18:27:01.663429 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.663449    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd689351-fd99-4084-a584-dd4fac79625f-istio-csr-ca-configmap\") pod \"cd689351-fd99-4084-a584-dd4fac79625f\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") "
Sep 16 18:27:01.663756 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.663493    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd689351-fd99-4084-a584-dd4fac79625f-local-certs\") pod \"cd689351-fd99-4084-a584-dd4fac79625f\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") "
Sep 16 18:27:01.663756 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.663556    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xwgt\" (UniqueName: \"kubernetes.io/projected/cd689351-fd99-4084-a584-dd4fac79625f-kube-api-access-7xwgt\") pod \"cd689351-fd99-4084-a584-dd4fac79625f\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") "
Sep 16 18:27:01.663756 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.663583    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd689351-fd99-4084-a584-dd4fac79625f-istio-token\") pod \"cd689351-fd99-4084-a584-dd4fac79625f\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") "
Sep 16 18:27:01.663756 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.663623    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-istio-kubeconfig\") pod \"cd689351-fd99-4084-a584-dd4fac79625f\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") "
Sep 16 18:27:01.663756 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.663663    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-cacerts\") pod \"cd689351-fd99-4084-a584-dd4fac79625f\" (UID: \"cd689351-fd99-4084-a584-dd4fac79625f\") "
Sep 16 18:27:01.664003 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.663969    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd689351-fd99-4084-a584-dd4fac79625f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "cd689351-fd99-4084-a584-dd4fac79625f" (UID: "cd689351-fd99-4084-a584-dd4fac79625f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:27:01.665989 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.665958    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd689351-fd99-4084-a584-dd4fac79625f-local-certs" (OuterVolumeSpecName: "local-certs") pod "cd689351-fd99-4084-a584-dd4fac79625f" (UID: "cd689351-fd99-4084-a584-dd4fac79625f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:01.666163 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.666124    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd689351-fd99-4084-a584-dd4fac79625f-kube-api-access-7xwgt" (OuterVolumeSpecName: "kube-api-access-7xwgt") pod "cd689351-fd99-4084-a584-dd4fac79625f" (UID: "cd689351-fd99-4084-a584-dd4fac79625f"). InnerVolumeSpecName "kube-api-access-7xwgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:27:01.666163 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.666140    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "cd689351-fd99-4084-a584-dd4fac79625f" (UID: "cd689351-fd99-4084-a584-dd4fac79625f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:27:01.666334 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.666196    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd689351-fd99-4084-a584-dd4fac79625f-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd689351-fd99-4084-a584-dd4fac79625f" (UID: "cd689351-fd99-4084-a584-dd4fac79625f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:27:01.666472 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.666450    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "cd689351-fd99-4084-a584-dd4fac79625f" (UID: "cd689351-fd99-4084-a584-dd4fac79625f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:27:01.666472 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.666463    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-cacerts" (OuterVolumeSpecName: "cacerts") pod "cd689351-fd99-4084-a584-dd4fac79625f" (UID: "cd689351-fd99-4084-a584-dd4fac79625f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:27:01.764953 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.764918    2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-istio-kubeconfig\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:27:01.764953 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.764945    2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-cacerts\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:27:01.764953 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.764954    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cd689351-fd99-4084-a584-dd4fac79625f-istio-csr-dns-cert\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:27:01.765169 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.764963    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cd689351-fd99-4084-a584-dd4fac79625f-istio-csr-ca-configmap\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:27:01.765169 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.764973    2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd689351-fd99-4084-a584-dd4fac79625f-local-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:27:01.765169 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.764981    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xwgt\" (UniqueName: \"kubernetes.io/projected/cd689351-fd99-4084-a584-dd4fac79625f-kube-api-access-7xwgt\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:27:01.765169 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:01.764989    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd689351-fd99-4084-a584-dd4fac79625f-istio-token\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:27:02.326491 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:02.326376    2562 generic.go:356] "Generic (PLEG): container finished" podID="cd689351-fd99-4084-a584-dd4fac79625f" containerID="a4929848b868b31997842c0012c07d0112f51e3c1474b8aadbcd792959c1a7c8" exitCode=0
Sep 16 18:27:02.326491 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:02.326413    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk" event={"ID":"cd689351-fd99-4084-a584-dd4fac79625f","Type":"ContainerDied","Data":"a4929848b868b31997842c0012c07d0112f51e3c1474b8aadbcd792959c1a7c8"}
Sep 16 18:27:02.326491 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:02.326439    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"
Sep 16 18:27:02.326491 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:02.326474    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk" event={"ID":"cd689351-fd99-4084-a584-dd4fac79625f","Type":"ContainerDied","Data":"6a070b803e38c7ac6be970075b198a3cf22b1082e4386338f6123a8449c922aa"}
Sep 16 18:27:02.327178 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:02.326503    2562 scope.go:117] "RemoveContainer" containerID="a4929848b868b31997842c0012c07d0112f51e3c1474b8aadbcd792959c1a7c8"
Sep 16 18:27:02.329155 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:02.329126    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9" event={"ID":"0f0b145a-64ef-4406-ad4b-23d6357b5ec2","Type":"ContainerStarted","Data":"a99b7d7ce4b934acc41089ef79d600fac1924fb2eacc4bd85f6b5b2bc9a1fe48"}
Sep 16 18:27:02.334760 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:02.334685    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk" event={"ID":"51297c36-21d1-48af-8f30-df5470ab7218","Type":"ContainerStarted","Data":"0d13596d44562ebb5eb19da7d88b9f98f39e7c55ac969dbf876424c3261ebf51"}
Sep 16 18:27:02.344663 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:02.344629    2562 scope.go:117] "RemoveContainer" containerID="a4929848b868b31997842c0012c07d0112f51e3c1474b8aadbcd792959c1a7c8"
Sep 16 18:27:02.345165 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:27:02.345134    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4929848b868b31997842c0012c07d0112f51e3c1474b8aadbcd792959c1a7c8\": container with ID starting with a4929848b868b31997842c0012c07d0112f51e3c1474b8aadbcd792959c1a7c8 not found: ID does not exist" containerID="a4929848b868b31997842c0012c07d0112f51e3c1474b8aadbcd792959c1a7c8"
Sep 16 18:27:02.345257 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:02.345180    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4929848b868b31997842c0012c07d0112f51e3c1474b8aadbcd792959c1a7c8"} err="failed to get container status \"a4929848b868b31997842c0012c07d0112f51e3c1474b8aadbcd792959c1a7c8\": rpc error: code = NotFound desc = could not find container \"a4929848b868b31997842c0012c07d0112f51e3c1474b8aadbcd792959c1a7c8\": container with ID starting with a4929848b868b31997842c0012c07d0112f51e3c1474b8aadbcd792959c1a7c8 not found: ID does not exist"
Sep 16 18:27:02.359386 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:02.359356    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"]
Sep 16 18:27:02.365578 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:02.365523    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-s5vnk"]
Sep 16 18:27:02.550403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:02.550370    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd689351-fd99-4084-a584-dd4fac79625f" path="/var/lib/kubelet/pods/cd689351-fd99-4084-a584-dd4fac79625f/volumes"
Sep 16 18:27:04.351051 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:04.351011    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk" event={"ID":"51297c36-21d1-48af-8f30-df5470ab7218","Type":"ContainerStarted","Data":"74f899d3dc8d111fbd07211f0f3dd909e915d09cf76687c9d425765fc9b875ed"}
Sep 16 18:27:04.351530 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:04.351123    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk"
Sep 16 18:27:04.369957 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:04.369899    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk" podStartSLOduration=1.915173783 podStartE2EDuration="4.369881707s" totalImagesPullingTime="2.454707924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:27:04.369642645 +0000 UTC m=+662.234546693" watchObservedRunningTime="2026-09-16 18:27:04.369881707 +0000 UTC m=+662.234785758"
Sep 16 18:27:05.357952 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:05.357911    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9" event={"ID":"0f0b145a-64ef-4406-ad4b-23d6357b5ec2","Type":"ContainerStarted","Data":"ae1b17541b6badda97c11f562e42b3cbfbf589213178190e917b6000594d2508"}
Sep 16 18:27:05.358430 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:05.358164    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9"
Sep 16 18:27:05.377049 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:05.376994    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9" podStartSLOduration=2.242039505 podStartE2EDuration="5.376976577s" totalImagesPullingTime="3.134937072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:27:05.375793354 +0000 UTC m=+663.240697403" watchObservedRunningTime="2026-09-16 18:27:05.376976577 +0000 UTC m=+663.241880624"
Sep 16 18:27:32.652346 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:32.652294    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:32.714412 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:32.714358    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:35.363818 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:35.363786    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-mmvjk"
Sep 16 18:27:36.364878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:36.364847    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r2lj9"
Sep 16 18:27:44.069716 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:44.069645    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:45.483821 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:45.483779    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:50.750721 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:50.750664    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:50.853659 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:50.853601    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:54.665065 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.665030    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"]
Sep 16 18:27:54.665416 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.665324    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd689351-fd99-4084-a584-dd4fac79625f" containerName="discovery"
Sep 16 18:27:54.665416 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.665335    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd689351-fd99-4084-a584-dd4fac79625f" containerName="discovery"
Sep 16 18:27:54.665416 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.665404    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd689351-fd99-4084-a584-dd4fac79625f" containerName="discovery"
Sep 16 18:27:54.670290 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.670271    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"
Sep 16 18:27:54.672914 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.672890    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-28c5h\""
Sep 16 18:27:54.673016 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.672946    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:27:54.673949 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.673934    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:27:54.677082 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.677059    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"]
Sep 16 18:27:54.754817 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.754780    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"]
Sep 16 18:27:54.759528 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.759490    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"
Sep 16 18:27:54.768439 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.768413    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"]
Sep 16 18:27:54.811254 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.811215    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/727563ad-41be-4abd-93cd-069fb1a26297-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5\" (UID: \"727563ad-41be-4abd-93cd-069fb1a26297\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"
Sep 16 18:27:54.811463 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.811280    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/727563ad-41be-4abd-93cd-069fb1a26297-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5\" (UID: \"727563ad-41be-4abd-93cd-069fb1a26297\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"
Sep 16 18:27:54.811463 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.811330    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qksd4\" (UniqueName: \"kubernetes.io/projected/727563ad-41be-4abd-93cd-069fb1a26297-kube-api-access-qksd4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5\" (UID: \"727563ad-41be-4abd-93cd-069fb1a26297\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"
Sep 16 18:27:54.863181 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.863147    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"]
Sep 16 18:27:54.866346 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.866326    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"
Sep 16 18:27:54.878960 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.878939    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"]
Sep 16 18:27:54.912496 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.912464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/727563ad-41be-4abd-93cd-069fb1a26297-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5\" (UID: \"727563ad-41be-4abd-93cd-069fb1a26297\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"
Sep 16 18:27:54.912663 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.912505    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bctxf\" (UniqueName: \"kubernetes.io/projected/d49f44e0-6df5-49b6-a849-b58dd805f77d-kube-api-access-bctxf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582\" (UID: \"d49f44e0-6df5-49b6-a849-b58dd805f77d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"
Sep 16 18:27:54.912663 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.912533    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qksd4\" (UniqueName: \"kubernetes.io/projected/727563ad-41be-4abd-93cd-069fb1a26297-kube-api-access-qksd4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5\" (UID: \"727563ad-41be-4abd-93cd-069fb1a26297\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"
Sep 16 18:27:54.912751 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.912680    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d49f44e0-6df5-49b6-a849-b58dd805f77d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582\" (UID: \"d49f44e0-6df5-49b6-a849-b58dd805f77d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"
Sep 16 18:27:54.912751 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.912733    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/727563ad-41be-4abd-93cd-069fb1a26297-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5\" (UID: \"727563ad-41be-4abd-93cd-069fb1a26297\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"
Sep 16 18:27:54.912823 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.912754    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d49f44e0-6df5-49b6-a849-b58dd805f77d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582\" (UID: \"d49f44e0-6df5-49b6-a849-b58dd805f77d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"
Sep 16 18:27:54.912903 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.912882    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/727563ad-41be-4abd-93cd-069fb1a26297-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5\" (UID: \"727563ad-41be-4abd-93cd-069fb1a26297\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"
Sep 16 18:27:54.913057 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.913040    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/727563ad-41be-4abd-93cd-069fb1a26297-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5\" (UID: \"727563ad-41be-4abd-93cd-069fb1a26297\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"
Sep 16 18:27:54.921489 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.921425    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qksd4\" (UniqueName: \"kubernetes.io/projected/727563ad-41be-4abd-93cd-069fb1a26297-kube-api-access-qksd4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5\" (UID: \"727563ad-41be-4abd-93cd-069fb1a26297\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"
Sep 16 18:27:54.960948 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.960923    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"]
Sep 16 18:27:54.963445 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.963429    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"
Sep 16 18:27:54.974863 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.974842    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"]
Sep 16 18:27:54.980285 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:54.980268    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"
Sep 16 18:27:55.013596 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.013535    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d49f44e0-6df5-49b6-a849-b58dd805f77d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582\" (UID: \"d49f44e0-6df5-49b6-a849-b58dd805f77d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"
Sep 16 18:27:55.013764 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.013613    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s\" (UID: \"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"
Sep 16 18:27:55.013764 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.013650    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmb4q\" (UniqueName: \"kubernetes.io/projected/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-kube-api-access-zmb4q\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s\" (UID: \"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"
Sep 16 18:27:55.013764 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.013687    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s\" (UID: \"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"
Sep 16 18:27:55.013919 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.013787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bctxf\" (UniqueName: \"kubernetes.io/projected/d49f44e0-6df5-49b6-a849-b58dd805f77d-kube-api-access-bctxf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582\" (UID: \"d49f44e0-6df5-49b6-a849-b58dd805f77d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"
Sep 16 18:27:55.013919 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.013883    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d49f44e0-6df5-49b6-a849-b58dd805f77d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582\" (UID: \"d49f44e0-6df5-49b6-a849-b58dd805f77d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"
Sep 16 18:27:55.013919 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.013890    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d49f44e0-6df5-49b6-a849-b58dd805f77d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582\" (UID: \"d49f44e0-6df5-49b6-a849-b58dd805f77d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"
Sep 16 18:27:55.014208 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.014185    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d49f44e0-6df5-49b6-a849-b58dd805f77d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582\" (UID: \"d49f44e0-6df5-49b6-a849-b58dd805f77d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"
Sep 16 18:27:55.023034 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.023005    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bctxf\" (UniqueName: \"kubernetes.io/projected/d49f44e0-6df5-49b6-a849-b58dd805f77d-kube-api-access-bctxf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582\" (UID: \"d49f44e0-6df5-49b6-a849-b58dd805f77d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"
Sep 16 18:27:55.070290 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.070254    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"
Sep 16 18:27:55.115323 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.115280    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/343990d4-42d8-4e19-b536-8bcd706ee49d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44\" (UID: \"343990d4-42d8-4e19-b536-8bcd706ee49d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"
Sep 16 18:27:55.115413 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.115337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s\" (UID: \"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"
Sep 16 18:27:55.115413 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.115380    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/343990d4-42d8-4e19-b536-8bcd706ee49d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44\" (UID: \"343990d4-42d8-4e19-b536-8bcd706ee49d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"
Sep 16 18:27:55.115482 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.115412    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmb4q\" (UniqueName: \"kubernetes.io/projected/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-kube-api-access-zmb4q\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s\" (UID: \"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"
Sep 16 18:27:55.115532 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.115505    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s\" (UID: \"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"
Sep 16 18:27:55.115712 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.115582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79xcc\" (UniqueName: \"kubernetes.io/projected/343990d4-42d8-4e19-b536-8bcd706ee49d-kube-api-access-79xcc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44\" (UID: \"343990d4-42d8-4e19-b536-8bcd706ee49d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"
Sep 16 18:27:55.116001 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.115981    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s\" (UID: \"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"
Sep 16 18:27:55.116064 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.116016    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s\" (UID: \"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"
Sep 16 18:27:55.124885 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.124855    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmb4q\" (UniqueName: \"kubernetes.io/projected/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-kube-api-access-zmb4q\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s\" (UID: \"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"
Sep 16 18:27:55.175693 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.175603    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"
Sep 16 18:27:55.216421 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.216385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/343990d4-42d8-4e19-b536-8bcd706ee49d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44\" (UID: \"343990d4-42d8-4e19-b536-8bcd706ee49d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"
Sep 16 18:27:55.216581 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.216431    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/343990d4-42d8-4e19-b536-8bcd706ee49d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44\" (UID: \"343990d4-42d8-4e19-b536-8bcd706ee49d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"
Sep 16 18:27:55.216581 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.216480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xcc\" (UniqueName: \"kubernetes.io/projected/343990d4-42d8-4e19-b536-8bcd706ee49d-kube-api-access-79xcc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44\" (UID: \"343990d4-42d8-4e19-b536-8bcd706ee49d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"
Sep 16 18:27:55.216868 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.216846    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/343990d4-42d8-4e19-b536-8bcd706ee49d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44\" (UID: \"343990d4-42d8-4e19-b536-8bcd706ee49d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"
Sep 16 18:27:55.216910 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.216862    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/343990d4-42d8-4e19-b536-8bcd706ee49d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44\" (UID: \"343990d4-42d8-4e19-b536-8bcd706ee49d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"
Sep 16 18:27:55.226336 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.226196    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79xcc\" (UniqueName: \"kubernetes.io/projected/343990d4-42d8-4e19-b536-8bcd706ee49d-kube-api-access-79xcc\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44\" (UID: \"343990d4-42d8-4e19-b536-8bcd706ee49d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"
Sep 16 18:27:55.254591 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.254564    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"]
Sep 16 18:27:55.256798 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:27:55.256771    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727563ad_41be_4abd_93cd_069fb1a26297.slice/crio-c1da4816222bc012145b08ef0dba06144c585b666bc09a52d3d79f9c73545fd0 WatchSource:0}: Error finding container c1da4816222bc012145b08ef0dba06144c585b666bc09a52d3d79f9c73545fd0: Status 404 returned error can't find the container with id c1da4816222bc012145b08ef0dba06144c585b666bc09a52d3d79f9c73545fd0
Sep 16 18:27:55.272483 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.272451    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"
Sep 16 18:27:55.342110 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.342082    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"]
Sep 16 18:27:55.345156 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:27:55.345122    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49f44e0_6df5_49b6_a849_b58dd805f77d.slice/crio-2fb56bf9272ffde6102cb2a952e5aa15a451290dda3baefd98680c9dd45772c2 WatchSource:0}: Error finding container 2fb56bf9272ffde6102cb2a952e5aa15a451290dda3baefd98680c9dd45772c2: Status 404 returned error can't find the container with id 2fb56bf9272ffde6102cb2a952e5aa15a451290dda3baefd98680c9dd45772c2
Sep 16 18:27:55.449467 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.449443    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"]
Sep 16 18:27:55.451004 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:27:55.450978    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa3ef6d_7907_4c9b_b898_5a79cb4651c8.slice/crio-4a2e3a3a8ea952d0eafefa049c8c8acc8c834744c90534f38e5f6a451aa858fb WatchSource:0}: Error finding container 4a2e3a3a8ea952d0eafefa049c8c8acc8c834744c90534f38e5f6a451aa858fb: Status 404 returned error can't find the container with id 4a2e3a3a8ea952d0eafefa049c8c8acc8c834744c90534f38e5f6a451aa858fb
Sep 16 18:27:55.543057 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.543029    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"]
Sep 16 18:27:55.579320 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:27:55.579292    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343990d4_42d8_4e19_b536_8bcd706ee49d.slice/crio-4d1fd757b08e0060f0012c265d9eeceeab7619f09f66eed352570ddf07678849 WatchSource:0}: Error finding container 4d1fd757b08e0060f0012c265d9eeceeab7619f09f66eed352570ddf07678849: Status 404 returned error can't find the container with id 4d1fd757b08e0060f0012c265d9eeceeab7619f09f66eed352570ddf07678849
Sep 16 18:27:55.587893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.587863    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44" event={"ID":"343990d4-42d8-4e19-b536-8bcd706ee49d","Type":"ContainerStarted","Data":"4d1fd757b08e0060f0012c265d9eeceeab7619f09f66eed352570ddf07678849"}
Sep 16 18:27:55.589272 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.589247    2562 generic.go:356] "Generic (PLEG): container finished" podID="d49f44e0-6df5-49b6-a849-b58dd805f77d" containerID="ef454dfcfc87cdda728a60f7b8d0f9bf9d89fded7cb136037478795578240e42" exitCode=0
Sep 16 18:27:55.589369 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.589322    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582" event={"ID":"d49f44e0-6df5-49b6-a849-b58dd805f77d","Type":"ContainerDied","Data":"ef454dfcfc87cdda728a60f7b8d0f9bf9d89fded7cb136037478795578240e42"}
Sep 16 18:27:55.589369 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.589363    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582" event={"ID":"d49f44e0-6df5-49b6-a849-b58dd805f77d","Type":"ContainerStarted","Data":"2fb56bf9272ffde6102cb2a952e5aa15a451290dda3baefd98680c9dd45772c2"}
Sep 16 18:27:55.590913 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.590889    2562 generic.go:356] "Generic (PLEG): container finished" podID="727563ad-41be-4abd-93cd-069fb1a26297" containerID="75164af851cbe373197b9f11557967b0c7c76a5f33778303db64e3f1aa5de2a9" exitCode=0
Sep 16 18:27:55.591012 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.590916    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5" event={"ID":"727563ad-41be-4abd-93cd-069fb1a26297","Type":"ContainerDied","Data":"75164af851cbe373197b9f11557967b0c7c76a5f33778303db64e3f1aa5de2a9"}
Sep 16 18:27:55.591012 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.590964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5" event={"ID":"727563ad-41be-4abd-93cd-069fb1a26297","Type":"ContainerStarted","Data":"c1da4816222bc012145b08ef0dba06144c585b666bc09a52d3d79f9c73545fd0"}
Sep 16 18:27:55.592845 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.592822    2562 generic.go:356] "Generic (PLEG): container finished" podID="4aa3ef6d-7907-4c9b-b898-5a79cb4651c8" containerID="193eb796229fb8d4022887054fae8e75b2f706a4f57a2933d7e9fb8cb77d7f0c" exitCode=0
Sep 16 18:27:55.592952 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.592900    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s" event={"ID":"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8","Type":"ContainerDied","Data":"193eb796229fb8d4022887054fae8e75b2f706a4f57a2933d7e9fb8cb77d7f0c"}
Sep 16 18:27:55.592952 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:55.592943    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s" event={"ID":"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8","Type":"ContainerStarted","Data":"4a2e3a3a8ea952d0eafefa049c8c8acc8c834744c90534f38e5f6a451aa858fb"}
Sep 16 18:27:56.599484 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:56.599377    2562 generic.go:356] "Generic (PLEG): container finished" podID="4aa3ef6d-7907-4c9b-b898-5a79cb4651c8" containerID="ff5de5878d818f53c8687b93aef2fd053057af57e13a73823b28079390b76d92" exitCode=0
Sep 16 18:27:56.599484 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:56.599466    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s" event={"ID":"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8","Type":"ContainerDied","Data":"ff5de5878d818f53c8687b93aef2fd053057af57e13a73823b28079390b76d92"}
Sep 16 18:27:56.601451 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:56.601313    2562 generic.go:356] "Generic (PLEG): container finished" podID="343990d4-42d8-4e19-b536-8bcd706ee49d" containerID="90f07c1221a749584da9cf2dd6bbf7214f0f6ae685207af1446a67f3860d1c60" exitCode=0
Sep 16 18:27:56.601451 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:56.601437    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44" event={"ID":"343990d4-42d8-4e19-b536-8bcd706ee49d","Type":"ContainerDied","Data":"90f07c1221a749584da9cf2dd6bbf7214f0f6ae685207af1446a67f3860d1c60"}
Sep 16 18:27:56.603878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:56.603852    2562 generic.go:356] "Generic (PLEG): container finished" podID="727563ad-41be-4abd-93cd-069fb1a26297" containerID="d64b8683b91ce39752750a23e63f1f59e4005a209e58e81578d2120a349ec056" exitCode=0
Sep 16 18:27:56.603983 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:56.603881    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5" event={"ID":"727563ad-41be-4abd-93cd-069fb1a26297","Type":"ContainerDied","Data":"d64b8683b91ce39752750a23e63f1f59e4005a209e58e81578d2120a349ec056"}
Sep 16 18:27:57.616432 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:57.616406    2562 generic.go:356] "Generic (PLEG): container finished" podID="4aa3ef6d-7907-4c9b-b898-5a79cb4651c8" containerID="5485fc420fbc639b061909eb34a39798ed33ccea59ad44c54c119136dfa3c413" exitCode=0
Sep 16 18:27:57.616893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:57.616482    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s" event={"ID":"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8","Type":"ContainerDied","Data":"5485fc420fbc639b061909eb34a39798ed33ccea59ad44c54c119136dfa3c413"}
Sep 16 18:27:57.618238 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:57.618212    2562 generic.go:356] "Generic (PLEG): container finished" podID="d49f44e0-6df5-49b6-a849-b58dd805f77d" containerID="6b065ce8aa46a2cc5e9cc6fbc91ce8260a3beb6ddfe278041662495dd9b7f6d2" exitCode=0
Sep 16 18:27:57.618328 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:57.618291    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582" event={"ID":"d49f44e0-6df5-49b6-a849-b58dd805f77d","Type":"ContainerDied","Data":"6b065ce8aa46a2cc5e9cc6fbc91ce8260a3beb6ddfe278041662495dd9b7f6d2"}
Sep 16 18:27:57.620375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:57.620355    2562 generic.go:356] "Generic (PLEG): container finished" podID="727563ad-41be-4abd-93cd-069fb1a26297" containerID="6ce0d1faa76cb1af9f7fe80a183802cab04f88677d3f70af3f9b28ae6111cd15" exitCode=0
Sep 16 18:27:57.620472 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:57.620391    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5" event={"ID":"727563ad-41be-4abd-93cd-069fb1a26297","Type":"ContainerDied","Data":"6ce0d1faa76cb1af9f7fe80a183802cab04f88677d3f70af3f9b28ae6111cd15"}
Sep 16 18:27:58.628000 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.627969    2562 generic.go:356] "Generic (PLEG): container finished" podID="343990d4-42d8-4e19-b536-8bcd706ee49d" containerID="b73579710f5574253897ffa13518923921795f979c3c83a4ae6a6a95e0b7b0e0" exitCode=0
Sep 16 18:27:58.628414 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.628040    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44" event={"ID":"343990d4-42d8-4e19-b536-8bcd706ee49d","Type":"ContainerDied","Data":"b73579710f5574253897ffa13518923921795f979c3c83a4ae6a6a95e0b7b0e0"}
Sep 16 18:27:58.630251 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.630230    2562 generic.go:356] "Generic (PLEG): container finished" podID="d49f44e0-6df5-49b6-a849-b58dd805f77d" containerID="9d4d0841fd2b03581f66b38d611c01e77ca626351f31774fc777bf64a5586094" exitCode=0
Sep 16 18:27:58.630344 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.630285    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582" event={"ID":"d49f44e0-6df5-49b6-a849-b58dd805f77d","Type":"ContainerDied","Data":"9d4d0841fd2b03581f66b38d611c01e77ca626351f31774fc777bf64a5586094"}
Sep 16 18:27:58.840250 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.840220    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"
Sep 16 18:27:58.843126 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.843104    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"
Sep 16 18:27:58.944794 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.944715    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/727563ad-41be-4abd-93cd-069fb1a26297-util\") pod \"727563ad-41be-4abd-93cd-069fb1a26297\" (UID: \"727563ad-41be-4abd-93cd-069fb1a26297\") "
Sep 16 18:27:58.944794 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.944751    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/727563ad-41be-4abd-93cd-069fb1a26297-bundle\") pod \"727563ad-41be-4abd-93cd-069fb1a26297\" (UID: \"727563ad-41be-4abd-93cd-069fb1a26297\") "
Sep 16 18:27:58.944794 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.944788    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmb4q\" (UniqueName: \"kubernetes.io/projected/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-kube-api-access-zmb4q\") pod \"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8\" (UID: \"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8\") "
Sep 16 18:27:58.945061 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.944809    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qksd4\" (UniqueName: \"kubernetes.io/projected/727563ad-41be-4abd-93cd-069fb1a26297-kube-api-access-qksd4\") pod \"727563ad-41be-4abd-93cd-069fb1a26297\" (UID: \"727563ad-41be-4abd-93cd-069fb1a26297\") "
Sep 16 18:27:58.945061 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.944874    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-util\") pod \"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8\" (UID: \"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8\") "
Sep 16 18:27:58.945061 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.944899    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-bundle\") pod \"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8\" (UID: \"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8\") "
Sep 16 18:27:58.945513 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.945448    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-bundle" (OuterVolumeSpecName: "bundle") pod "4aa3ef6d-7907-4c9b-b898-5a79cb4651c8" (UID: "4aa3ef6d-7907-4c9b-b898-5a79cb4651c8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:58.945647 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.945608    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727563ad-41be-4abd-93cd-069fb1a26297-bundle" (OuterVolumeSpecName: "bundle") pod "727563ad-41be-4abd-93cd-069fb1a26297" (UID: "727563ad-41be-4abd-93cd-069fb1a26297"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:58.947049 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.947023    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-kube-api-access-zmb4q" (OuterVolumeSpecName: "kube-api-access-zmb4q") pod "4aa3ef6d-7907-4c9b-b898-5a79cb4651c8" (UID: "4aa3ef6d-7907-4c9b-b898-5a79cb4651c8"). InnerVolumeSpecName "kube-api-access-zmb4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:27:58.947126 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.947060    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727563ad-41be-4abd-93cd-069fb1a26297-kube-api-access-qksd4" (OuterVolumeSpecName: "kube-api-access-qksd4") pod "727563ad-41be-4abd-93cd-069fb1a26297" (UID: "727563ad-41be-4abd-93cd-069fb1a26297"). InnerVolumeSpecName "kube-api-access-qksd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:27:58.950199 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.950164    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-util" (OuterVolumeSpecName: "util") pod "4aa3ef6d-7907-4c9b-b898-5a79cb4651c8" (UID: "4aa3ef6d-7907-4c9b-b898-5a79cb4651c8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:58.951091 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:58.951071    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727563ad-41be-4abd-93cd-069fb1a26297-util" (OuterVolumeSpecName: "util") pod "727563ad-41be-4abd-93cd-069fb1a26297" (UID: "727563ad-41be-4abd-93cd-069fb1a26297"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:59.046427 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.046383    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmb4q\" (UniqueName: \"kubernetes.io/projected/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-kube-api-access-zmb4q\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:27:59.046427 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.046420    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qksd4\" (UniqueName: \"kubernetes.io/projected/727563ad-41be-4abd-93cd-069fb1a26297-kube-api-access-qksd4\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:27:59.046427 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.046431    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-util\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:27:59.046677 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.046441    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4aa3ef6d-7907-4c9b-b898-5a79cb4651c8-bundle\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:27:59.046677 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.046449    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/727563ad-41be-4abd-93cd-069fb1a26297-util\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:27:59.046677 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.046456    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/727563ad-41be-4abd-93cd-069fb1a26297-bundle\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:27:59.637505 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.637474    2562 generic.go:356] "Generic (PLEG): container finished" podID="343990d4-42d8-4e19-b536-8bcd706ee49d" containerID="361e84576f8fe223ac5ce45ca1a17d3f99384d6b991ceee438240c1655631ba6" exitCode=0
Sep 16 18:27:59.637965 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.637581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44" event={"ID":"343990d4-42d8-4e19-b536-8bcd706ee49d","Type":"ContainerDied","Data":"361e84576f8fe223ac5ce45ca1a17d3f99384d6b991ceee438240c1655631ba6"}
Sep 16 18:27:59.639432 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.639414    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5"
Sep 16 18:27:59.639556 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.639429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad7kl5" event={"ID":"727563ad-41be-4abd-93cd-069fb1a26297","Type":"ContainerDied","Data":"c1da4816222bc012145b08ef0dba06144c585b666bc09a52d3d79f9c73545fd0"}
Sep 16 18:27:59.639556 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.639466    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1da4816222bc012145b08ef0dba06144c585b666bc09a52d3d79f9c73545fd0"
Sep 16 18:27:59.641532 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.641516    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s"
Sep 16 18:27:59.641648 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.641566    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64829s" event={"ID":"4aa3ef6d-7907-4c9b-b898-5a79cb4651c8","Type":"ContainerDied","Data":"4a2e3a3a8ea952d0eafefa049c8c8acc8c834744c90534f38e5f6a451aa858fb"}
Sep 16 18:27:59.641648 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.641619    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a2e3a3a8ea952d0eafefa049c8c8acc8c834744c90534f38e5f6a451aa858fb"
Sep 16 18:27:59.778171 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.778142    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"
Sep 16 18:27:59.853010 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.852969    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d49f44e0-6df5-49b6-a849-b58dd805f77d-util\") pod \"d49f44e0-6df5-49b6-a849-b58dd805f77d\" (UID: \"d49f44e0-6df5-49b6-a849-b58dd805f77d\") "
Sep 16 18:27:59.853190 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.853034    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bctxf\" (UniqueName: \"kubernetes.io/projected/d49f44e0-6df5-49b6-a849-b58dd805f77d-kube-api-access-bctxf\") pod \"d49f44e0-6df5-49b6-a849-b58dd805f77d\" (UID: \"d49f44e0-6df5-49b6-a849-b58dd805f77d\") "
Sep 16 18:27:59.853190 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.853074    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d49f44e0-6df5-49b6-a849-b58dd805f77d-bundle\") pod \"d49f44e0-6df5-49b6-a849-b58dd805f77d\" (UID: \"d49f44e0-6df5-49b6-a849-b58dd805f77d\") "
Sep 16 18:27:59.853681 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.853649    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49f44e0-6df5-49b6-a849-b58dd805f77d-bundle" (OuterVolumeSpecName: "bundle") pod "d49f44e0-6df5-49b6-a849-b58dd805f77d" (UID: "d49f44e0-6df5-49b6-a849-b58dd805f77d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:59.855272 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.855248    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49f44e0-6df5-49b6-a849-b58dd805f77d-kube-api-access-bctxf" (OuterVolumeSpecName: "kube-api-access-bctxf") pod "d49f44e0-6df5-49b6-a849-b58dd805f77d" (UID: "d49f44e0-6df5-49b6-a849-b58dd805f77d"). InnerVolumeSpecName "kube-api-access-bctxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:27:59.857814 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.857779    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49f44e0-6df5-49b6-a849-b58dd805f77d-util" (OuterVolumeSpecName: "util") pod "d49f44e0-6df5-49b6-a849-b58dd805f77d" (UID: "d49f44e0-6df5-49b6-a849-b58dd805f77d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:59.954355 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.954302    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d49f44e0-6df5-49b6-a849-b58dd805f77d-util\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:27:59.954355 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.954346    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bctxf\" (UniqueName: \"kubernetes.io/projected/d49f44e0-6df5-49b6-a849-b58dd805f77d-kube-api-access-bctxf\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:27:59.954632 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:27:59.954362    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d49f44e0-6df5-49b6-a849-b58dd805f77d-bundle\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:28:00.653236 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:00.653200    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582"
Sep 16 18:28:00.653236 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:00.653217    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbb582" event={"ID":"d49f44e0-6df5-49b6-a849-b58dd805f77d","Type":"ContainerDied","Data":"2fb56bf9272ffde6102cb2a952e5aa15a451290dda3baefd98680c9dd45772c2"}
Sep 16 18:28:00.653709 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:00.653298    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fb56bf9272ffde6102cb2a952e5aa15a451290dda3baefd98680c9dd45772c2"
Sep 16 18:28:00.803291 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:00.803258    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"
Sep 16 18:28:00.859579 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:00.859477    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/343990d4-42d8-4e19-b536-8bcd706ee49d-bundle\") pod \"343990d4-42d8-4e19-b536-8bcd706ee49d\" (UID: \"343990d4-42d8-4e19-b536-8bcd706ee49d\") "
Sep 16 18:28:00.859579 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:00.859537    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/343990d4-42d8-4e19-b536-8bcd706ee49d-util\") pod \"343990d4-42d8-4e19-b536-8bcd706ee49d\" (UID: \"343990d4-42d8-4e19-b536-8bcd706ee49d\") "
Sep 16 18:28:00.859805 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:00.859700    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79xcc\" (UniqueName: \"kubernetes.io/projected/343990d4-42d8-4e19-b536-8bcd706ee49d-kube-api-access-79xcc\") pod \"343990d4-42d8-4e19-b536-8bcd706ee49d\" (UID: \"343990d4-42d8-4e19-b536-8bcd706ee49d\") "
Sep 16 18:28:00.860050 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:00.860026    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343990d4-42d8-4e19-b536-8bcd706ee49d-bundle" (OuterVolumeSpecName: "bundle") pod "343990d4-42d8-4e19-b536-8bcd706ee49d" (UID: "343990d4-42d8-4e19-b536-8bcd706ee49d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:28:00.861761 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:00.861731    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343990d4-42d8-4e19-b536-8bcd706ee49d-kube-api-access-79xcc" (OuterVolumeSpecName: "kube-api-access-79xcc") pod "343990d4-42d8-4e19-b536-8bcd706ee49d" (UID: "343990d4-42d8-4e19-b536-8bcd706ee49d"). InnerVolumeSpecName "kube-api-access-79xcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:28:00.961019 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:00.960979    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79xcc\" (UniqueName: \"kubernetes.io/projected/343990d4-42d8-4e19-b536-8bcd706ee49d-kube-api-access-79xcc\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:28:00.961019 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:00.961012    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/343990d4-42d8-4e19-b536-8bcd706ee49d-bundle\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:28:01.016677 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:01.016641    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343990d4-42d8-4e19-b536-8bcd706ee49d-util" (OuterVolumeSpecName: "util") pod "343990d4-42d8-4e19-b536-8bcd706ee49d" (UID: "343990d4-42d8-4e19-b536-8bcd706ee49d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:28:01.062301 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:01.062266    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/343990d4-42d8-4e19-b536-8bcd706ee49d-util\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:28:01.660361 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:01.660329    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44" event={"ID":"343990d4-42d8-4e19-b536-8bcd706ee49d","Type":"ContainerDied","Data":"4d1fd757b08e0060f0012c265d9eeceeab7619f09f66eed352570ddf07678849"}
Sep 16 18:28:01.660361 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:01.660349    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r9w44"
Sep 16 18:28:01.660891 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:01.660373    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d1fd757b08e0060f0012c265d9eeceeab7619f09f66eed352570ddf07678849"
Sep 16 18:28:06.809160 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:06.809125    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:08.871748 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.871709    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xx6jl"]
Sep 16 18:28:08.872128 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872054    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa3ef6d-7907-4c9b-b898-5a79cb4651c8" containerName="extract"
Sep 16 18:28:08.872128 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872066    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa3ef6d-7907-4c9b-b898-5a79cb4651c8" containerName="extract"
Sep 16 18:28:08.872128 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872078    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343990d4-42d8-4e19-b536-8bcd706ee49d" containerName="util"
Sep 16 18:28:08.872128 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872084    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="343990d4-42d8-4e19-b536-8bcd706ee49d" containerName="util"
Sep 16 18:28:08.872128 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872092    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49f44e0-6df5-49b6-a849-b58dd805f77d" containerName="extract"
Sep 16 18:28:08.872128 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872097    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49f44e0-6df5-49b6-a849-b58dd805f77d" containerName="extract"
Sep 16 18:28:08.872128 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872106    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49f44e0-6df5-49b6-a849-b58dd805f77d" containerName="util"
Sep 16 18:28:08.872128 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872115    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49f44e0-6df5-49b6-a849-b58dd805f77d" containerName="util"
Sep 16 18:28:08.872128 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872124    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343990d4-42d8-4e19-b536-8bcd706ee49d" containerName="extract"
Sep 16 18:28:08.872128 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872129    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="343990d4-42d8-4e19-b536-8bcd706ee49d" containerName="extract"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872141    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727563ad-41be-4abd-93cd-069fb1a26297" containerName="util"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872146    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="727563ad-41be-4abd-93cd-069fb1a26297" containerName="util"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872151    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49f44e0-6df5-49b6-a849-b58dd805f77d" containerName="pull"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872156    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49f44e0-6df5-49b6-a849-b58dd805f77d" containerName="pull"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872163    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="343990d4-42d8-4e19-b536-8bcd706ee49d" containerName="pull"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872168    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="343990d4-42d8-4e19-b536-8bcd706ee49d" containerName="pull"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872174    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727563ad-41be-4abd-93cd-069fb1a26297" containerName="pull"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872179    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="727563ad-41be-4abd-93cd-069fb1a26297" containerName="pull"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872186    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa3ef6d-7907-4c9b-b898-5a79cb4651c8" containerName="pull"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872194    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa3ef6d-7907-4c9b-b898-5a79cb4651c8" containerName="pull"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872207    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727563ad-41be-4abd-93cd-069fb1a26297" containerName="extract"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872212    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="727563ad-41be-4abd-93cd-069fb1a26297" containerName="extract"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872221    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa3ef6d-7907-4c9b-b898-5a79cb4651c8" containerName="util"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872225    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa3ef6d-7907-4c9b-b898-5a79cb4651c8" containerName="util"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872283    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49f44e0-6df5-49b6-a849-b58dd805f77d" containerName="extract"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872293    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aa3ef6d-7907-4c9b-b898-5a79cb4651c8" containerName="extract"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872299    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="727563ad-41be-4abd-93cd-069fb1a26297" containerName="extract"
Sep 16 18:28:08.872420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.872307    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="343990d4-42d8-4e19-b536-8bcd706ee49d" containerName="extract"
Sep 16 18:28:08.875287 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.875270    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xx6jl"
Sep 16 18:28:08.881005 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.880979    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:28:08.881294 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.881274    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 18:28:08.881433 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.881413    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:28:08.881500 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.881453    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fx9v4\""
Sep 16 18:28:08.890135 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.890105    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xx6jl"]
Sep 16 18:28:08.922234 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:08.922202    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscww\" (UniqueName: \"kubernetes.io/projected/824ea996-86ae-4433-8d3c-13b9dcfbf498-kube-api-access-kscww\") pod \"dns-operator-controller-manager-7b84cd7c9c-xx6jl\" (UID: \"824ea996-86ae-4433-8d3c-13b9dcfbf498\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xx6jl"
Sep 16 18:28:09.022884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:09.022848    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscww\" (UniqueName: \"kubernetes.io/projected/824ea996-86ae-4433-8d3c-13b9dcfbf498-kube-api-access-kscww\") pod \"dns-operator-controller-manager-7b84cd7c9c-xx6jl\" (UID: \"824ea996-86ae-4433-8d3c-13b9dcfbf498\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xx6jl"
Sep 16 18:28:09.033168 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:09.033137    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscww\" (UniqueName: \"kubernetes.io/projected/824ea996-86ae-4433-8d3c-13b9dcfbf498-kube-api-access-kscww\") pod \"dns-operator-controller-manager-7b84cd7c9c-xx6jl\" (UID: \"824ea996-86ae-4433-8d3c-13b9dcfbf498\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xx6jl"
Sep 16 18:28:09.185235 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:09.185151    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xx6jl"
Sep 16 18:28:09.208640 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:09.208612    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:09.467180 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:09.467153    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xx6jl"]
Sep 16 18:28:09.468788 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:28:09.468763    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824ea996_86ae_4433_8d3c_13b9dcfbf498.slice/crio-1673c3a2737d6a2f2aebe4d6410b71a5953e2c632c9bf359a910608165949c0e WatchSource:0}: Error finding container 1673c3a2737d6a2f2aebe4d6410b71a5953e2c632c9bf359a910608165949c0e: Status 404 returned error can't find the container with id 1673c3a2737d6a2f2aebe4d6410b71a5953e2c632c9bf359a910608165949c0e
Sep 16 18:28:09.608387 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:09.608353    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:09.702614 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:09.702569    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xx6jl" event={"ID":"824ea996-86ae-4433-8d3c-13b9dcfbf498","Type":"ContainerStarted","Data":"1673c3a2737d6a2f2aebe4d6410b71a5953e2c632c9bf359a910608165949c0e"}
Sep 16 18:28:11.041086 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:11.041039    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:11.395163 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:11.395083    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-chp8t"]
Sep 16 18:28:11.401535 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:11.401508    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-chp8t"
Sep 16 18:28:11.405871 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:11.405837    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-tgfbr\""
Sep 16 18:28:11.418787 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:11.418760    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-chp8t"]
Sep 16 18:28:11.444734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:11.444698    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9cgn\" (UniqueName: \"kubernetes.io/projected/6d1bf43f-7668-4f7d-93c4-2a0ba5f7744b-kube-api-access-p9cgn\") pod \"authorino-operator-76b478bd78-chp8t\" (UID: \"6d1bf43f-7668-4f7d-93c4-2a0ba5f7744b\") " pod="kuadrant-system/authorino-operator-76b478bd78-chp8t"
Sep 16 18:28:11.545658 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:11.545610    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9cgn\" (UniqueName: \"kubernetes.io/projected/6d1bf43f-7668-4f7d-93c4-2a0ba5f7744b-kube-api-access-p9cgn\") pod \"authorino-operator-76b478bd78-chp8t\" (UID: \"6d1bf43f-7668-4f7d-93c4-2a0ba5f7744b\") " pod="kuadrant-system/authorino-operator-76b478bd78-chp8t"
Sep 16 18:28:11.559532 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:11.559468    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9cgn\" (UniqueName: \"kubernetes.io/projected/6d1bf43f-7668-4f7d-93c4-2a0ba5f7744b-kube-api-access-p9cgn\") pod \"authorino-operator-76b478bd78-chp8t\" (UID: \"6d1bf43f-7668-4f7d-93c4-2a0ba5f7744b\") " pod="kuadrant-system/authorino-operator-76b478bd78-chp8t"
Sep 16 18:28:11.715108 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:11.714985    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-chp8t"
Sep 16 18:28:12.009493 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:12.009373    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:12.258926 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:12.258894    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:12.316441 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:12.316413    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-chp8t"]
Sep 16 18:28:12.318182 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:28:12.318157    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1bf43f_7668_4f7d_93c4_2a0ba5f7744b.slice/crio-cf6cbdcb03d1371f7377228d8024b6e48fc5001bdb4cb1f84675daae40d59545 WatchSource:0}: Error finding container cf6cbdcb03d1371f7377228d8024b6e48fc5001bdb4cb1f84675daae40d59545: Status 404 returned error can't find the container with id cf6cbdcb03d1371f7377228d8024b6e48fc5001bdb4cb1f84675daae40d59545
Sep 16 18:28:12.424166 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:12.424131    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:12.722167 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:12.722128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-chp8t" event={"ID":"6d1bf43f-7668-4f7d-93c4-2a0ba5f7744b","Type":"ContainerStarted","Data":"cf6cbdcb03d1371f7377228d8024b6e48fc5001bdb4cb1f84675daae40d59545"}
Sep 16 18:28:12.723696 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:12.723671    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xx6jl" event={"ID":"824ea996-86ae-4433-8d3c-13b9dcfbf498","Type":"ContainerStarted","Data":"1a94eeab595696ddc82566a7145ad7c5c1d250ce7fbf4e354a3a8dcc20c8a63a"}
Sep 16 18:28:12.723825 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:12.723799    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xx6jl"
Sep 16 18:28:12.744847 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:12.744800    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xx6jl" podStartSLOduration=2.150012333 podStartE2EDuration="4.744785475s" totalImagesPullingTime="2.594773142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:12.742501003 +0000 UTC m=+730.607405050" watchObservedRunningTime="2026-09-16 18:28:12.744785475 +0000 UTC m=+730.609689523"
Sep 16 18:28:13.008962 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:13.008880    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:13.850160 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:13.850131    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:13.950424 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:13.950392    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:14.735554 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:14.735512    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-chp8t" event={"ID":"6d1bf43f-7668-4f7d-93c4-2a0ba5f7744b","Type":"ContainerStarted","Data":"8cba42bf8bf328db16a80907656167e6e45bc9edc3220a5f3b03f56f8b8f6d2e"}
Sep 16 18:28:14.735736 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:14.735643    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-chp8t"
Sep 16 18:28:14.753168 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:14.753110    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-chp8t" podStartSLOduration=1.626211337 podStartE2EDuration="3.753096463s" totalImagesPullingTime="2.126885126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:14.751446897 +0000 UTC m=+732.616350946" watchObservedRunningTime="2026-09-16 18:28:14.753096463 +0000 UTC m=+732.618000512"
Sep 16 18:28:15.905559 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:15.905504    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"]
Sep 16 18:28:15.908142 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:15.908123    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:15.910473 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:15.910455    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dfm9g\""
Sep 16 18:28:15.924364 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:15.924336    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"]
Sep 16 18:28:15.984304 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:15.984263    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd67c493-2f45-4882-a995-854eda6ec4e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kr2c6\" (UID: \"cd67c493-2f45-4882-a995-854eda6ec4e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:15.984304 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:15.984304    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng59g\" (UniqueName: \"kubernetes.io/projected/cd67c493-2f45-4882-a995-854eda6ec4e8-kube-api-access-ng59g\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kr2c6\" (UID: \"cd67c493-2f45-4882-a995-854eda6ec4e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:16.085615 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:16.085532    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd67c493-2f45-4882-a995-854eda6ec4e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kr2c6\" (UID: \"cd67c493-2f45-4882-a995-854eda6ec4e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:16.085615 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:16.085621    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng59g\" (UniqueName: \"kubernetes.io/projected/cd67c493-2f45-4882-a995-854eda6ec4e8-kube-api-access-ng59g\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kr2c6\" (UID: \"cd67c493-2f45-4882-a995-854eda6ec4e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:16.085965 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:16.085930    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd67c493-2f45-4882-a995-854eda6ec4e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kr2c6\" (UID: \"cd67c493-2f45-4882-a995-854eda6ec4e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:16.095026 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:16.094998    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng59g\" (UniqueName: \"kubernetes.io/projected/cd67c493-2f45-4882-a995-854eda6ec4e8-kube-api-access-ng59g\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kr2c6\" (UID: \"cd67c493-2f45-4882-a995-854eda6ec4e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:16.217618 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:16.217580    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:16.414248 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:16.414211    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:16.495764 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:16.495737    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"]
Sep 16 18:28:16.498456 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:16.498420    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:16.498685 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:28:16.498660    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd67c493_2f45_4882_a995_854eda6ec4e8.slice/crio-17f90562cd654fcbdde6dca41fb2c09ccf681908dd005cf80f0f690ce7458ee9 WatchSource:0}: Error finding container 17f90562cd654fcbdde6dca41fb2c09ccf681908dd005cf80f0f690ce7458ee9: Status 404 returned error can't find the container with id 17f90562cd654fcbdde6dca41fb2c09ccf681908dd005cf80f0f690ce7458ee9
Sep 16 18:28:16.748141 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:16.748056    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6" event={"ID":"cd67c493-2f45-4882-a995-854eda6ec4e8","Type":"ContainerStarted","Data":"17f90562cd654fcbdde6dca41fb2c09ccf681908dd005cf80f0f690ce7458ee9"}
Sep 16 18:28:18.203799 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:18.203764    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"]
Sep 16 18:28:18.206588 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:18.206569    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"
Sep 16 18:28:18.209074 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:18.209046    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kd427\""
Sep 16 18:28:18.227194 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:18.227154    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"]
Sep 16 18:28:18.305098 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:18.305057    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmnlz\" (UniqueName: \"kubernetes.io/projected/3825b90b-6a0c-4af7-8048-293009d34edd-kube-api-access-qmnlz\") pod \"limitador-operator-controller-manager-74495c69f8-q2mvc\" (UID: \"3825b90b-6a0c-4af7-8048-293009d34edd\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"
Sep 16 18:28:18.406045 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:18.406002    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnlz\" (UniqueName: \"kubernetes.io/projected/3825b90b-6a0c-4af7-8048-293009d34edd-kube-api-access-qmnlz\") pod \"limitador-operator-controller-manager-74495c69f8-q2mvc\" (UID: \"3825b90b-6a0c-4af7-8048-293009d34edd\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"
Sep 16 18:28:18.421992 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:18.421962    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnlz\" (UniqueName: \"kubernetes.io/projected/3825b90b-6a0c-4af7-8048-293009d34edd-kube-api-access-qmnlz\") pod \"limitador-operator-controller-manager-74495c69f8-q2mvc\" (UID: \"3825b90b-6a0c-4af7-8048-293009d34edd\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"
Sep 16 18:28:18.517027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:18.516944    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"
Sep 16 18:28:18.787930 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:18.787877    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"]
Sep 16 18:28:18.790366 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:28:18.790342    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3825b90b_6a0c_4af7_8048_293009d34edd.slice/crio-87202691d8240ba0957df50c6a5834a4a69eea09b929b7204678eb934737c384 WatchSource:0}: Error finding container 87202691d8240ba0957df50c6a5834a4a69eea09b929b7204678eb934737c384: Status 404 returned error can't find the container with id 87202691d8240ba0957df50c6a5834a4a69eea09b929b7204678eb934737c384
Sep 16 18:28:19.438535 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:19.438488    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:19.534715 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:19.534655    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:19.769513 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:19.769387    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc" event={"ID":"3825b90b-6a0c-4af7-8048-293009d34edd","Type":"ContainerStarted","Data":"87202691d8240ba0957df50c6a5834a4a69eea09b929b7204678eb934737c384"}
Sep 16 18:28:20.777573 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:20.777521    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc" event={"ID":"3825b90b-6a0c-4af7-8048-293009d34edd","Type":"ContainerStarted","Data":"f92cfc11760d8b1400a7827c0be211c9d5526dda7459c22ff50419d8fea1dedd"}
Sep 16 18:28:20.778002 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:20.777608    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"
Sep 16 18:28:20.794736 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:20.794599    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc" podStartSLOduration=1.242729837 podStartE2EDuration="2.794580032s" totalImagesPullingTime="1.551850195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:20.793938613 +0000 UTC m=+738.658842663" watchObservedRunningTime="2026-09-16 18:28:20.794580032 +0000 UTC m=+738.659484082"
Sep 16 18:28:22.790286 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:22.790253    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6" event={"ID":"cd67c493-2f45-4882-a995-854eda6ec4e8","Type":"ContainerStarted","Data":"b3d765a7ee0829ec808e6a333f792ac75164d8ee69d586093b1c3205e76509f8"}
Sep 16 18:28:22.790709 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:22.790384    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:22.813069 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:22.813020    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6" podStartSLOduration=2.185144327 podStartE2EDuration="7.813005644s" totalImagesPullingTime="5.627861317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:22.810418505 +0000 UTC m=+740.675322552" watchObservedRunningTime="2026-09-16 18:28:22.813005644 +0000 UTC m=+740.677909689"
Sep 16 18:28:23.730499 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:23.730467    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-xx6jl"
Sep 16 18:28:24.624380 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:24.624346    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:24.730606 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:24.730571    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:24.824783 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:24.824750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:24.925777 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:24.925702    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:24.989455 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:24.989427    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:25.126271 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:25.126240    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:25.744876 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:25.744844    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-chp8t"
Sep 16 18:28:28.981866 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:28.981838    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:29.079811 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:29.079772    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:31.785978 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:31.785950    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"
Sep 16 18:28:33.798317 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:33.798282    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:34.943487 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:34.943452    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt"]
Sep 16 18:28:34.946789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:34.946767    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt"
Sep 16 18:28:34.966914 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:34.966882    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt"]
Sep 16 18:28:35.051019 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.050981    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1988dd6-58e3-4497-a46c-84043ed64a99-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-dk2tt\" (UID: \"f1988dd6-58e3-4497-a46c-84043ed64a99\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt"
Sep 16 18:28:35.051019 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.051019    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcbdk\" (UniqueName: \"kubernetes.io/projected/f1988dd6-58e3-4497-a46c-84043ed64a99-kube-api-access-zcbdk\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-dk2tt\" (UID: \"f1988dd6-58e3-4497-a46c-84043ed64a99\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt"
Sep 16 18:28:35.064779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.064745    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt"]
Sep 16 18:28:35.065018 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:28:35.064998    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-zcbdk], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt" podUID="f1988dd6-58e3-4497-a46c-84043ed64a99"
Sep 16 18:28:35.152308 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.152267    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1988dd6-58e3-4497-a46c-84043ed64a99-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-dk2tt\" (UID: \"f1988dd6-58e3-4497-a46c-84043ed64a99\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt"
Sep 16 18:28:35.152308 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.152310    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcbdk\" (UniqueName: \"kubernetes.io/projected/f1988dd6-58e3-4497-a46c-84043ed64a99-kube-api-access-zcbdk\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-dk2tt\" (UID: \"f1988dd6-58e3-4497-a46c-84043ed64a99\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt"
Sep 16 18:28:35.152743 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.152720    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1988dd6-58e3-4497-a46c-84043ed64a99-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-dk2tt\" (UID: \"f1988dd6-58e3-4497-a46c-84043ed64a99\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt"
Sep 16 18:28:35.163560 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.163520    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcbdk\" (UniqueName: \"kubernetes.io/projected/f1988dd6-58e3-4497-a46c-84043ed64a99-kube-api-access-zcbdk\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-dk2tt\" (UID: \"f1988dd6-58e3-4497-a46c-84043ed64a99\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt"
Sep 16 18:28:35.869399 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.869359    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt"
Sep 16 18:28:35.875634 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.875611    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt"
Sep 16 18:28:35.919773 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.919731    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt"]
Sep 16 18:28:35.927209 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.927185    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt"]
Sep 16 18:28:35.939101 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.939076    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"]
Sep 16 18:28:35.939306 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.939283    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6" podUID="cd67c493-2f45-4882-a995-854eda6ec4e8" containerName="manager" containerID="cri-o://b3d765a7ee0829ec808e6a333f792ac75164d8ee69d586093b1c3205e76509f8" gracePeriod=2
Sep 16 18:28:35.945193 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.945166    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"]
Sep 16 18:28:35.959191 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.958962    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1988dd6-58e3-4497-a46c-84043ed64a99-extensions-socket-volume\") pod \"f1988dd6-58e3-4497-a46c-84043ed64a99\" (UID: \"f1988dd6-58e3-4497-a46c-84043ed64a99\") "
Sep 16 18:28:35.959191 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.959051    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcbdk\" (UniqueName: \"kubernetes.io/projected/f1988dd6-58e3-4497-a46c-84043ed64a99-kube-api-access-zcbdk\") pod \"f1988dd6-58e3-4497-a46c-84043ed64a99\" (UID: \"f1988dd6-58e3-4497-a46c-84043ed64a99\") "
Sep 16 18:28:35.959382 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.959244    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1988dd6-58e3-4497-a46c-84043ed64a99-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f1988dd6-58e3-4497-a46c-84043ed64a99" (UID: "f1988dd6-58e3-4497-a46c-84043ed64a99"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:28:35.965318 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.965291    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"]
Sep 16 18:28:35.965688 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.965563    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc" podUID="3825b90b-6a0c-4af7-8048-293009d34edd" containerName="manager" containerID="cri-o://f92cfc11760d8b1400a7827c0be211c9d5526dda7459c22ff50419d8fea1dedd" gracePeriod=2
Sep 16 18:28:35.969914 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.969883    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1988dd6-58e3-4497-a46c-84043ed64a99-kube-api-access-zcbdk" (OuterVolumeSpecName: "kube-api-access-zcbdk") pod "f1988dd6-58e3-4497-a46c-84043ed64a99" (UID: "f1988dd6-58e3-4497-a46c-84043ed64a99"). InnerVolumeSpecName "kube-api-access-zcbdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:28:35.971182 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.971160    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"]
Sep 16 18:28:35.971624 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.971608    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd67c493-2f45-4882-a995-854eda6ec4e8" containerName="manager"
Sep 16 18:28:35.971679 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.971625    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd67c493-2f45-4882-a995-854eda6ec4e8" containerName="manager"
Sep 16 18:28:35.971716 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.971692    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd67c493-2f45-4882-a995-854eda6ec4e8" containerName="manager"
Sep 16 18:28:35.973771 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.973754    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"
Sep 16 18:28:35.979465 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.979440    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"]
Sep 16 18:28:35.994805 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.994772    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"]
Sep 16 18:28:35.995892 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:28:35.995732    2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3825b90b-6a0c-4af7-8048-293009d34edd\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-q2mvc\": pods \"limitador-operator-controller-manager-74495c69f8-q2mvc\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"
Sep 16 18:28:35.997918 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.997901    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vfphc"]
Sep 16 18:28:35.998203 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.998185    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3825b90b-6a0c-4af7-8048-293009d34edd" containerName="manager"
Sep 16 18:28:35.998203 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.998197    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3825b90b-6a0c-4af7-8048-293009d34edd" containerName="manager"
Sep 16 18:28:35.998364 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:35.998262    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3825b90b-6a0c-4af7-8048-293009d34edd" containerName="manager"
Sep 16 18:28:35.999115 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:28:35.999093    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kr2c6\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" podUID="cd67c493-2f45-4882-a995-854eda6ec4e8" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:36.000159 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.000140    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vfphc"
Sep 16 18:28:36.001017 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:28:36.000988    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kr2c6\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" podUID="cd67c493-2f45-4882-a995-854eda6ec4e8" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:36.020279 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.020253    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vfphc"]
Sep 16 18:28:36.027492 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:28:36.027458    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-q2mvc\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" podUID="3825b90b-6a0c-4af7-8048-293009d34edd" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"
Sep 16 18:28:36.029434 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:28:36.029407    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kr2c6\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" podUID="cd67c493-2f45-4882-a995-854eda6ec4e8" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:36.031294 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:28:36.031271    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-q2mvc\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" podUID="3825b90b-6a0c-4af7-8048-293009d34edd" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"
Sep 16 18:28:36.060196 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.060168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p45r\" (UniqueName: \"kubernetes.io/projected/2f9fc572-cef6-4865-aa1a-58824223d510-kube-api-access-2p45r\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6d5zb\" (UID: \"2f9fc572-cef6-4865-aa1a-58824223d510\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"
Sep 16 18:28:36.060369 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.060233    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f9fc572-cef6-4865-aa1a-58824223d510-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6d5zb\" (UID: \"2f9fc572-cef6-4865-aa1a-58824223d510\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"
Sep 16 18:28:36.060438 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.060360    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1988dd6-58e3-4497-a46c-84043ed64a99-extensions-socket-volume\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:28:36.060438 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.060383    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcbdk\" (UniqueName: \"kubernetes.io/projected/f1988dd6-58e3-4497-a46c-84043ed64a99-kube-api-access-zcbdk\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:28:36.161262 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.161216    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8b7v\" (UniqueName: \"kubernetes.io/projected/14542f6e-df6c-4aa1-82e1-fc7602e560ab-kube-api-access-s8b7v\") pod \"limitador-operator-controller-manager-74495c69f8-vfphc\" (UID: \"14542f6e-df6c-4aa1-82e1-fc7602e560ab\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vfphc"
Sep 16 18:28:36.161736 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.161699    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f9fc572-cef6-4865-aa1a-58824223d510-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6d5zb\" (UID: \"2f9fc572-cef6-4865-aa1a-58824223d510\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"
Sep 16 18:28:36.162174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.161858    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p45r\" (UniqueName: \"kubernetes.io/projected/2f9fc572-cef6-4865-aa1a-58824223d510-kube-api-access-2p45r\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6d5zb\" (UID: \"2f9fc572-cef6-4865-aa1a-58824223d510\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"
Sep 16 18:28:36.162174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.162089    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f9fc572-cef6-4865-aa1a-58824223d510-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6d5zb\" (UID: \"2f9fc572-cef6-4865-aa1a-58824223d510\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"
Sep 16 18:28:36.185847 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.185818    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p45r\" (UniqueName: \"kubernetes.io/projected/2f9fc572-cef6-4865-aa1a-58824223d510-kube-api-access-2p45r\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6d5zb\" (UID: \"2f9fc572-cef6-4865-aa1a-58824223d510\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"
Sep 16 18:28:36.211588 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.211536    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:36.214643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.214621    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"
Sep 16 18:28:36.214793 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:28:36.214770    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-q2mvc\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" podUID="3825b90b-6a0c-4af7-8048-293009d34edd" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"
Sep 16 18:28:36.218368 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:28:36.218347    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kr2c6\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" podUID="cd67c493-2f45-4882-a995-854eda6ec4e8" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:36.220406 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:28:36.220389    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-q2mvc\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" podUID="3825b90b-6a0c-4af7-8048-293009d34edd" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"
Sep 16 18:28:36.223528 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:28:36.223505    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kr2c6\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" podUID="cd67c493-2f45-4882-a995-854eda6ec4e8" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:36.229839 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.229810    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:36.263490 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.263453    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8b7v\" (UniqueName: \"kubernetes.io/projected/14542f6e-df6c-4aa1-82e1-fc7602e560ab-kube-api-access-s8b7v\") pod \"limitador-operator-controller-manager-74495c69f8-vfphc\" (UID: \"14542f6e-df6c-4aa1-82e1-fc7602e560ab\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vfphc"
Sep 16 18:28:36.289341 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.289302    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8b7v\" (UniqueName: \"kubernetes.io/projected/14542f6e-df6c-4aa1-82e1-fc7602e560ab-kube-api-access-s8b7v\") pod \"limitador-operator-controller-manager-74495c69f8-vfphc\" (UID: \"14542f6e-df6c-4aa1-82e1-fc7602e560ab\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vfphc"
Sep 16 18:28:36.346820 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.346784    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:36.363800 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.363771    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmnlz\" (UniqueName: \"kubernetes.io/projected/3825b90b-6a0c-4af7-8048-293009d34edd-kube-api-access-qmnlz\") pod \"3825b90b-6a0c-4af7-8048-293009d34edd\" (UID: \"3825b90b-6a0c-4af7-8048-293009d34edd\") "
Sep 16 18:28:36.363926 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.363817    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd67c493-2f45-4882-a995-854eda6ec4e8-extensions-socket-volume\") pod \"cd67c493-2f45-4882-a995-854eda6ec4e8\" (UID: \"cd67c493-2f45-4882-a995-854eda6ec4e8\") "
Sep 16 18:28:36.363926 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.363846    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng59g\" (UniqueName: \"kubernetes.io/projected/cd67c493-2f45-4882-a995-854eda6ec4e8-kube-api-access-ng59g\") pod \"cd67c493-2f45-4882-a995-854eda6ec4e8\" (UID: \"cd67c493-2f45-4882-a995-854eda6ec4e8\") "
Sep 16 18:28:36.364345 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.364311    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd67c493-2f45-4882-a995-854eda6ec4e8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cd67c493-2f45-4882-a995-854eda6ec4e8" (UID: "cd67c493-2f45-4882-a995-854eda6ec4e8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:28:36.365944 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.365921    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3825b90b-6a0c-4af7-8048-293009d34edd-kube-api-access-qmnlz" (OuterVolumeSpecName: "kube-api-access-qmnlz") pod "3825b90b-6a0c-4af7-8048-293009d34edd" (UID: "3825b90b-6a0c-4af7-8048-293009d34edd"). InnerVolumeSpecName "kube-api-access-qmnlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:28:36.366022 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.365999    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd67c493-2f45-4882-a995-854eda6ec4e8-kube-api-access-ng59g" (OuterVolumeSpecName: "kube-api-access-ng59g") pod "cd67c493-2f45-4882-a995-854eda6ec4e8" (UID: "cd67c493-2f45-4882-a995-854eda6ec4e8"). InnerVolumeSpecName "kube-api-access-ng59g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:28:36.385187 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.385119    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"
Sep 16 18:28:36.392572 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.392531    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vfphc"
Sep 16 18:28:36.465414 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.465376    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmnlz\" (UniqueName: \"kubernetes.io/projected/3825b90b-6a0c-4af7-8048-293009d34edd-kube-api-access-qmnlz\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:28:36.465591 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.465417    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cd67c493-2f45-4882-a995-854eda6ec4e8-extensions-socket-volume\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:28:36.465591 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.465435    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng59g\" (UniqueName: \"kubernetes.io/projected/cd67c493-2f45-4882-a995-854eda6ec4e8-kube-api-access-ng59g\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:28:36.550315 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.550278    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3825b90b-6a0c-4af7-8048-293009d34edd" path="/var/lib/kubelet/pods/3825b90b-6a0c-4af7-8048-293009d34edd/volumes"
Sep 16 18:28:36.550645 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.550632    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd67c493-2f45-4882-a995-854eda6ec4e8" path="/var/lib/kubelet/pods/cd67c493-2f45-4882-a995-854eda6ec4e8/volumes"
Sep 16 18:28:36.550941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.550929    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1988dd6-58e3-4497-a46c-84043ed64a99" path="/var/lib/kubelet/pods/f1988dd6-58e3-4497-a46c-84043ed64a99/volumes"
Sep 16 18:28:36.668569 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.668526    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"]
Sep 16 18:28:36.671460 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:28:36.671430    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9fc572_cef6_4865_aa1a_58824223d510.slice/crio-684841fd2a74d2ab16d8fe6887c378fcbdbdc73ae4dcd9b2a5c0d6aba3642e82 WatchSource:0}: Error finding container 684841fd2a74d2ab16d8fe6887c378fcbdbdc73ae4dcd9b2a5c0d6aba3642e82: Status 404 returned error can't find the container with id 684841fd2a74d2ab16d8fe6887c378fcbdbdc73ae4dcd9b2a5c0d6aba3642e82
Sep 16 18:28:36.682593 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.682572    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vfphc"]
Sep 16 18:28:36.684460 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:28:36.684435    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14542f6e_df6c_4aa1_82e1_fc7602e560ab.slice/crio-0dcc49b5331bdc4d5ada79bcdb656519129da344a9cb198bf6206982fee22170 WatchSource:0}: Error finding container 0dcc49b5331bdc4d5ada79bcdb656519129da344a9cb198bf6206982fee22170: Status 404 returned error can't find the container with id 0dcc49b5331bdc4d5ada79bcdb656519129da344a9cb198bf6206982fee22170
Sep 16 18:28:36.877712 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.877675    2562 generic.go:356] "Generic (PLEG): container finished" podID="3825b90b-6a0c-4af7-8048-293009d34edd" containerID="f92cfc11760d8b1400a7827c0be211c9d5526dda7459c22ff50419d8fea1dedd" exitCode=0
Sep 16 18:28:36.877924 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.877723    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2mvc"
Sep 16 18:28:36.877924 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.877779    2562 scope.go:117] "RemoveContainer" containerID="f92cfc11760d8b1400a7827c0be211c9d5526dda7459c22ff50419d8fea1dedd"
Sep 16 18:28:36.879851 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.879808    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vfphc" event={"ID":"14542f6e-df6c-4aa1-82e1-fc7602e560ab","Type":"ContainerStarted","Data":"8f9a04f18f4302c5fb17baa07d94b1b1a29a77fdb4284a7d0fe12bc883bcc7ca"}
Sep 16 18:28:36.879983 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.879875    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vfphc" event={"ID":"14542f6e-df6c-4aa1-82e1-fc7602e560ab","Type":"ContainerStarted","Data":"0dcc49b5331bdc4d5ada79bcdb656519129da344a9cb198bf6206982fee22170"}
Sep 16 18:28:36.880233 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.880046    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vfphc"
Sep 16 18:28:36.882017 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.881988    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb" event={"ID":"2f9fc572-cef6-4865-aa1a-58824223d510","Type":"ContainerStarted","Data":"51afd1bf353c11d055b8ec946595647d8365badd316095af00d397ab0fd2404c"}
Sep 16 18:28:36.882142 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.882042    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb" event={"ID":"2f9fc572-cef6-4865-aa1a-58824223d510","Type":"ContainerStarted","Data":"684841fd2a74d2ab16d8fe6887c378fcbdbdc73ae4dcd9b2a5c0d6aba3642e82"}
Sep 16 18:28:36.882428 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.882401    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"
Sep 16 18:28:36.883731 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.883711    2562 generic.go:356] "Generic (PLEG): container finished" podID="cd67c493-2f45-4882-a995-854eda6ec4e8" containerID="b3d765a7ee0829ec808e6a333f792ac75164d8ee69d586093b1c3205e76509f8" exitCode=0
Sep 16 18:28:36.883805 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.883746    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kr2c6"
Sep 16 18:28:36.883903 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.883882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-dk2tt"
Sep 16 18:28:36.888891 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.888876    2562 scope.go:117] "RemoveContainer" containerID="f92cfc11760d8b1400a7827c0be211c9d5526dda7459c22ff50419d8fea1dedd"
Sep 16 18:28:36.889135 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:28:36.889120    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f92cfc11760d8b1400a7827c0be211c9d5526dda7459c22ff50419d8fea1dedd\": container with ID starting with f92cfc11760d8b1400a7827c0be211c9d5526dda7459c22ff50419d8fea1dedd not found: ID does not exist" containerID="f92cfc11760d8b1400a7827c0be211c9d5526dda7459c22ff50419d8fea1dedd"
Sep 16 18:28:36.889195 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.889141    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f92cfc11760d8b1400a7827c0be211c9d5526dda7459c22ff50419d8fea1dedd"} err="failed to get container status \"f92cfc11760d8b1400a7827c0be211c9d5526dda7459c22ff50419d8fea1dedd\": rpc error: code = NotFound desc = could not find container \"f92cfc11760d8b1400a7827c0be211c9d5526dda7459c22ff50419d8fea1dedd\": container with ID starting with f92cfc11760d8b1400a7827c0be211c9d5526dda7459c22ff50419d8fea1dedd not found: ID does not exist"
Sep 16 18:28:36.889195 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.889157    2562 scope.go:117] "RemoveContainer" containerID="b3d765a7ee0829ec808e6a333f792ac75164d8ee69d586093b1c3205e76509f8"
Sep 16 18:28:36.897339 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.897324    2562 scope.go:117] "RemoveContainer" containerID="b3d765a7ee0829ec808e6a333f792ac75164d8ee69d586093b1c3205e76509f8"
Sep 16 18:28:36.897614 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:28:36.897589    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d765a7ee0829ec808e6a333f792ac75164d8ee69d586093b1c3205e76509f8\": container with ID starting with b3d765a7ee0829ec808e6a333f792ac75164d8ee69d586093b1c3205e76509f8 not found: ID does not exist" containerID="b3d765a7ee0829ec808e6a333f792ac75164d8ee69d586093b1c3205e76509f8"
Sep 16 18:28:36.897661 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.897622    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d765a7ee0829ec808e6a333f792ac75164d8ee69d586093b1c3205e76509f8"} err="failed to get container status \"b3d765a7ee0829ec808e6a333f792ac75164d8ee69d586093b1c3205e76509f8\": rpc error: code = NotFound desc = could not find container \"b3d765a7ee0829ec808e6a333f792ac75164d8ee69d586093b1c3205e76509f8\": container with ID starting with b3d765a7ee0829ec808e6a333f792ac75164d8ee69d586093b1c3205e76509f8 not found: ID does not exist"
Sep 16 18:28:36.901074 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.901035    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vfphc" podStartSLOduration=1.9010223609999999 podStartE2EDuration="1.901022361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:36.899759121 +0000 UTC m=+754.764663169" watchObservedRunningTime="2026-09-16 18:28:36.901022361 +0000 UTC m=+754.765926409"
Sep 16 18:28:36.920085 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:36.919987    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb" podStartSLOduration=1.919974603 podStartE2EDuration="1.919974603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:36.918420726 +0000 UTC m=+754.783324774" watchObservedRunningTime="2026-09-16 18:28:36.919974603 +0000 UTC m=+754.784878650"
Sep 16 18:28:38.692514 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:38.692461    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:38.799354 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:38.799317    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:44.083375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:44.083340    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:44.183665 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:44.183630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:44.435098 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:44.435011    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:44.536442 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:44.536381    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:47.893408 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:47.893373    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vfphc"
Sep 16 18:28:48.627146 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:48.627093    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:48.730591 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:48.730536    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:48.837455 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:48.837421    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:48.898473 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:48.898399    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"
Sep 16 18:28:51.273760 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.273726    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"]
Sep 16 18:28:51.274224 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.273912    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb" podUID="2f9fc572-cef6-4865-aa1a-58824223d510" containerName="manager" containerID="cri-o://51afd1bf353c11d055b8ec946595647d8365badd316095af00d397ab0fd2404c" gracePeriod=10
Sep 16 18:28:51.512870 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.512841    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"]
Sep 16 18:28:51.515188 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.515169    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"
Sep 16 18:28:51.520143 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.520121    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"
Sep 16 18:28:51.532537 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.532481    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"]
Sep 16 18:28:51.689983 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.689953    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p45r\" (UniqueName: \"kubernetes.io/projected/2f9fc572-cef6-4865-aa1a-58824223d510-kube-api-access-2p45r\") pod \"2f9fc572-cef6-4865-aa1a-58824223d510\" (UID: \"2f9fc572-cef6-4865-aa1a-58824223d510\") "
Sep 16 18:28:51.690158 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.690007    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f9fc572-cef6-4865-aa1a-58824223d510-extensions-socket-volume\") pod \"2f9fc572-cef6-4865-aa1a-58824223d510\" (UID: \"2f9fc572-cef6-4865-aa1a-58824223d510\") "
Sep 16 18:28:51.690158 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.690091    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bxgnx\" (UID: \"d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"
Sep 16 18:28:51.690158 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.690124    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gns7l\" (UniqueName: \"kubernetes.io/projected/d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce-kube-api-access-gns7l\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bxgnx\" (UID: \"d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"
Sep 16 18:28:51.690415 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.690392    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9fc572-cef6-4865-aa1a-58824223d510-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2f9fc572-cef6-4865-aa1a-58824223d510" (UID: "2f9fc572-cef6-4865-aa1a-58824223d510"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:28:51.692086 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.692054    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9fc572-cef6-4865-aa1a-58824223d510-kube-api-access-2p45r" (OuterVolumeSpecName: "kube-api-access-2p45r") pod "2f9fc572-cef6-4865-aa1a-58824223d510" (UID: "2f9fc572-cef6-4865-aa1a-58824223d510"). InnerVolumeSpecName "kube-api-access-2p45r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:28:51.791405 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.791307    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bxgnx\" (UID: \"d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"
Sep 16 18:28:51.791405 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.791358    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gns7l\" (UniqueName: \"kubernetes.io/projected/d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce-kube-api-access-gns7l\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bxgnx\" (UID: \"d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"
Sep 16 18:28:51.791704 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.791422    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p45r\" (UniqueName: \"kubernetes.io/projected/2f9fc572-cef6-4865-aa1a-58824223d510-kube-api-access-2p45r\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:28:51.791704 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.791433    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f9fc572-cef6-4865-aa1a-58824223d510-extensions-socket-volume\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:28:51.791819 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.791794    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bxgnx\" (UID: \"d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"
Sep 16 18:28:51.809194 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.809161    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gns7l\" (UniqueName: \"kubernetes.io/projected/d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce-kube-api-access-gns7l\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bxgnx\" (UID: \"d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"
Sep 16 18:28:51.826860 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.826833    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"
Sep 16 18:28:51.971168 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.971129    2562 generic.go:356] "Generic (PLEG): container finished" podID="2f9fc572-cef6-4865-aa1a-58824223d510" containerID="51afd1bf353c11d055b8ec946595647d8365badd316095af00d397ab0fd2404c" exitCode=0
Sep 16 18:28:51.971368 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.971180    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"
Sep 16 18:28:51.971368 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.971221    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb" event={"ID":"2f9fc572-cef6-4865-aa1a-58824223d510","Type":"ContainerDied","Data":"51afd1bf353c11d055b8ec946595647d8365badd316095af00d397ab0fd2404c"}
Sep 16 18:28:51.971368 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.971284    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb" event={"ID":"2f9fc572-cef6-4865-aa1a-58824223d510","Type":"ContainerDied","Data":"684841fd2a74d2ab16d8fe6887c378fcbdbdc73ae4dcd9b2a5c0d6aba3642e82"}
Sep 16 18:28:51.971368 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.971317    2562 scope.go:117] "RemoveContainer" containerID="51afd1bf353c11d055b8ec946595647d8365badd316095af00d397ab0fd2404c"
Sep 16 18:28:51.980229 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.980211    2562 scope.go:117] "RemoveContainer" containerID="51afd1bf353c11d055b8ec946595647d8365badd316095af00d397ab0fd2404c"
Sep 16 18:28:51.980496 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:28:51.980478    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51afd1bf353c11d055b8ec946595647d8365badd316095af00d397ab0fd2404c\": container with ID starting with 51afd1bf353c11d055b8ec946595647d8365badd316095af00d397ab0fd2404c not found: ID does not exist" containerID="51afd1bf353c11d055b8ec946595647d8365badd316095af00d397ab0fd2404c"
Sep 16 18:28:51.980580 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.980507    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51afd1bf353c11d055b8ec946595647d8365badd316095af00d397ab0fd2404c"} err="failed to get container status \"51afd1bf353c11d055b8ec946595647d8365badd316095af00d397ab0fd2404c\": rpc error: code = NotFound desc = could not find container \"51afd1bf353c11d055b8ec946595647d8365badd316095af00d397ab0fd2404c\": container with ID starting with 51afd1bf353c11d055b8ec946595647d8365badd316095af00d397ab0fd2404c not found: ID does not exist"
Sep 16 18:28:51.994056 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.994032    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"]
Sep 16 18:28:51.999260 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:51.999239    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6d5zb"]
Sep 16 18:28:52.101674 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:52.101600    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"]
Sep 16 18:28:52.104852 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:28:52.104830    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c53cd6_2a90_4e31_b726_dd79cdc5f4ce.slice/crio-c9308a35aa5eb660aa50b27f74d899ac619a9936a70fe4f37277159642ec9b2c WatchSource:0}: Error finding container c9308a35aa5eb660aa50b27f74d899ac619a9936a70fe4f37277159642ec9b2c: Status 404 returned error can't find the container with id c9308a35aa5eb660aa50b27f74d899ac619a9936a70fe4f37277159642ec9b2c
Sep 16 18:28:52.550964 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:52.550917    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f9fc572-cef6-4865-aa1a-58824223d510" path="/var/lib/kubelet/pods/2f9fc572-cef6-4865-aa1a-58824223d510/volumes"
Sep 16 18:28:52.978292 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:52.978247    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx" event={"ID":"d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce","Type":"ContainerStarted","Data":"0fd3499234483d013aadf94b87798104bedc354aa185aec447779a37826184ab"}
Sep 16 18:28:52.978497 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:52.978453    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"
Sep 16 18:28:52.978497 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:52.978481    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx" event={"ID":"d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce","Type":"ContainerStarted","Data":"c9308a35aa5eb660aa50b27f74d899ac619a9936a70fe4f37277159642ec9b2c"}
Sep 16 18:28:53.002957 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:53.002909    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx" podStartSLOduration=2.002894218 podStartE2EDuration="2.002894218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:52.999262062 +0000 UTC m=+770.864166134" watchObservedRunningTime="2026-09-16 18:28:53.002894218 +0000 UTC m=+770.867798265"
Sep 16 18:28:55.740598 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:55.740538    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:55.840698 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:55.840661    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:57.324600 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:57.324571    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:57.440243 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:28:57.440204    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:03.986279 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:03.986244    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"
Sep 16 18:29:07.571005 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.570968    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"]
Sep 16 18:29:07.571521 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.571312    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f9fc572-cef6-4865-aa1a-58824223d510" containerName="manager"
Sep 16 18:29:07.571521 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.571322    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f9fc572-cef6-4865-aa1a-58824223d510" containerName="manager"
Sep 16 18:29:07.571521 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.571385    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f9fc572-cef6-4865-aa1a-58824223d510" containerName="manager"
Sep 16 18:29:07.574463 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.574437    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.577413 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.577385    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dp69p\""
Sep 16 18:29:07.586640 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.586616    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"]
Sep 16 18:29:07.602690 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.602655    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:07.618383 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.618347    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.618383 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.618382    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.618622 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.618457    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-799mj\" (UniqueName: \"kubernetes.io/projected/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-kube-api-access-799mj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.618622 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.618490    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.618622 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.618506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.618622 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.618521    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.618622 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.618568    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.618795 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.618630    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.618795 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.618691    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.713518 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.713478    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:07.719679 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.719657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.719756 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.719714    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-799mj\" (UniqueName: \"kubernetes.io/projected/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-kube-api-access-799mj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.719756 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.719741    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.719840 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.719758    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.719840 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.719776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.719840 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.719799    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.719840 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.719823    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.720047 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.719860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.720115 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.720055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.720115 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.720100    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.720229 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.720141    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.720378 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.720342    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.720378 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.720376    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.720599 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.720582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.722192 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.722171    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.722325 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.722308    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.727227 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.727198    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.728171 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.728147    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-799mj\" (UniqueName: \"kubernetes.io/projected/2c8ffe4b-cb09-4d16-9740-7eba4c4d404f-kube-api-access-799mj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-2mcsv\" (UID: \"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:07.885430 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:07.885330    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:08.158801 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:08.158776    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"]
Sep 16 18:29:08.160319 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:29:08.160290    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8ffe4b_cb09_4d16_9740_7eba4c4d404f.slice/crio-20588691c2e740fe9f818893d065c2f1b5132a9c7e620e3ca30c5994c19d56b3 WatchSource:0}: Error finding container 20588691c2e740fe9f818893d065c2f1b5132a9c7e620e3ca30c5994c19d56b3: Status 404 returned error can't find the container with id 20588691c2e740fe9f818893d065c2f1b5132a9c7e620e3ca30c5994c19d56b3
Sep 16 18:29:08.162378 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:08.162347    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:29:08.162445 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:08.162432    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:29:08.162489 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:08.162469    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:29:09.069336 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:09.069298    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv" event={"ID":"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f","Type":"ContainerStarted","Data":"a65376a76981e132d5977015ef9b96ed1b88d576b667c4b1d0be4b640d513b1a"}
Sep 16 18:29:09.069775 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:09.069366    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv" event={"ID":"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f","Type":"ContainerStarted","Data":"20588691c2e740fe9f818893d065c2f1b5132a9c7e620e3ca30c5994c19d56b3"}
Sep 16 18:29:09.090250 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:09.090186    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv" podStartSLOduration=2.090168206 podStartE2EDuration="2.090168206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:09.087749141 +0000 UTC m=+786.952653188" watchObservedRunningTime="2026-09-16 18:29:09.090168206 +0000 UTC m=+786.955072324"
Sep 16 18:29:09.886066 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:09.886025    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:09.890675 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:09.890647    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:10.075056 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:10.075023    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:10.076084 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:10.076065    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:29:12.313166 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:12.313136    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:12.370943 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:12.370914    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:23.316346 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:23.316308    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:23.405071 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:23.405036    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:23.472389 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:23.472348    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:23.711990 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:23.711955    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:29:23.716407 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:23.716383    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-n726w"
Sep 16 18:29:23.718785 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:23.718757    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 18:29:23.718925 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:23.718903    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4l2gg\""
Sep 16 18:29:23.723006 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:23.722986    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:29:23.753854 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:23.753820    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:29:23.858352 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:23.858314    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7a958dd8-3400-445a-9733-bb5e512c7b44-config-file\") pod \"limitador-limitador-84994f7ffc-n726w\" (UID: \"7a958dd8-3400-445a-9733-bb5e512c7b44\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-n726w"
Sep 16 18:29:23.858535 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:23.858378    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khr6w\" (UniqueName: \"kubernetes.io/projected/7a958dd8-3400-445a-9733-bb5e512c7b44-kube-api-access-khr6w\") pod \"limitador-limitador-84994f7ffc-n726w\" (UID: \"7a958dd8-3400-445a-9733-bb5e512c7b44\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-n726w"
Sep 16 18:29:23.959787 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:23.959755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khr6w\" (UniqueName: \"kubernetes.io/projected/7a958dd8-3400-445a-9733-bb5e512c7b44-kube-api-access-khr6w\") pod \"limitador-limitador-84994f7ffc-n726w\" (UID: \"7a958dd8-3400-445a-9733-bb5e512c7b44\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-n726w"
Sep 16 18:29:23.959940 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:23.959824    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7a958dd8-3400-445a-9733-bb5e512c7b44-config-file\") pod \"limitador-limitador-84994f7ffc-n726w\" (UID: \"7a958dd8-3400-445a-9733-bb5e512c7b44\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-n726w"
Sep 16 18:29:23.960388 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:23.960369    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7a958dd8-3400-445a-9733-bb5e512c7b44-config-file\") pod \"limitador-limitador-84994f7ffc-n726w\" (UID: \"7a958dd8-3400-445a-9733-bb5e512c7b44\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-n726w"
Sep 16 18:29:23.967820 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:23.967756    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khr6w\" (UniqueName: \"kubernetes.io/projected/7a958dd8-3400-445a-9733-bb5e512c7b44-kube-api-access-khr6w\") pod \"limitador-limitador-84994f7ffc-n726w\" (UID: \"7a958dd8-3400-445a-9733-bb5e512c7b44\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-n726w"
Sep 16 18:29:24.027493 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:24.027460    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-n726w"
Sep 16 18:29:24.151412 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:24.151382    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:24.234318 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:24.234245    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:24.238779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:24.238751    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-pssg2"]
Sep 16 18:29:24.242491 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:24.242470    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-pssg2"
Sep 16 18:29:24.246000 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:24.245959    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-h6z7c\""
Sep 16 18:29:24.248466 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:24.248412    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-pssg2"]
Sep 16 18:29:24.302960 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:24.302934    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:29:24.304147 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:29:24.304113    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a958dd8_3400_445a_9733_bb5e512c7b44.slice/crio-1504835bd5a90268149e908cc6dd7bc06a6b81fc127531bc6447334728ecc1ee WatchSource:0}: Error finding container 1504835bd5a90268149e908cc6dd7bc06a6b81fc127531bc6447334728ecc1ee: Status 404 returned error can't find the container with id 1504835bd5a90268149e908cc6dd7bc06a6b81fc127531bc6447334728ecc1ee
Sep 16 18:29:24.363021 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:24.362981    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4jnv\" (UniqueName: \"kubernetes.io/projected/368bf6ba-c029-400b-830e-1ec83a9b2ef4-kube-api-access-q4jnv\") pod \"authorino-59c974bd48-pssg2\" (UID: \"368bf6ba-c029-400b-830e-1ec83a9b2ef4\") " pod="kuadrant-system/authorino-59c974bd48-pssg2"
Sep 16 18:29:24.464193 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:24.464159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4jnv\" (UniqueName: \"kubernetes.io/projected/368bf6ba-c029-400b-830e-1ec83a9b2ef4-kube-api-access-q4jnv\") pod \"authorino-59c974bd48-pssg2\" (UID: \"368bf6ba-c029-400b-830e-1ec83a9b2ef4\") " pod="kuadrant-system/authorino-59c974bd48-pssg2"
Sep 16 18:29:24.472478 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:24.472454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4jnv\" (UniqueName: \"kubernetes.io/projected/368bf6ba-c029-400b-830e-1ec83a9b2ef4-kube-api-access-q4jnv\") pod \"authorino-59c974bd48-pssg2\" (UID: \"368bf6ba-c029-400b-830e-1ec83a9b2ef4\") " pod="kuadrant-system/authorino-59c974bd48-pssg2"
Sep 16 18:29:24.552911 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:24.552837    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-pssg2"
Sep 16 18:29:24.825040 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:24.825015    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-pssg2"]
Sep 16 18:29:24.826974 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:29:24.826944    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod368bf6ba_c029_400b_830e_1ec83a9b2ef4.slice/crio-3e81ace2868b67a0cb14051e21b82a7ab6bd1136630f52cb602d9c1aeecbe633 WatchSource:0}: Error finding container 3e81ace2868b67a0cb14051e21b82a7ab6bd1136630f52cb602d9c1aeecbe633: Status 404 returned error can't find the container with id 3e81ace2868b67a0cb14051e21b82a7ab6bd1136630f52cb602d9c1aeecbe633
Sep 16 18:29:25.161249 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:25.161137    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-n726w" event={"ID":"7a958dd8-3400-445a-9733-bb5e512c7b44","Type":"ContainerStarted","Data":"1504835bd5a90268149e908cc6dd7bc06a6b81fc127531bc6447334728ecc1ee"}
Sep 16 18:29:25.162720 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:25.162689    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-pssg2" event={"ID":"368bf6ba-c029-400b-830e-1ec83a9b2ef4","Type":"ContainerStarted","Data":"3e81ace2868b67a0cb14051e21b82a7ab6bd1136630f52cb602d9c1aeecbe633"}
Sep 16 18:29:26.172580 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:26.172528    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-n726w" event={"ID":"7a958dd8-3400-445a-9733-bb5e512c7b44","Type":"ContainerStarted","Data":"7a859bccb172a9d070eabe772bc60fa39d134c3a2c4a7b5c0df88de7c4d8a72e"}
Sep 16 18:29:26.173019 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:26.172612    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-n726w"
Sep 16 18:29:26.188312 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:26.188261    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-n726w" podStartSLOduration=2.04594857 podStartE2EDuration="3.188245121s" totalImagesPullingTime="1.142296551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:26.18758179 +0000 UTC m=+804.052485840" watchObservedRunningTime="2026-09-16 18:29:26.188245121 +0000 UTC m=+804.053149222"
Sep 16 18:29:27.021643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:27.021610    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:27.114435 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:27.114404    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:27.179974 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:27.179936    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-pssg2" event={"ID":"368bf6ba-c029-400b-830e-1ec83a9b2ef4","Type":"ContainerStarted","Data":"074eecc3265244f95e4b3afca2317dbe86dd6f313e6aa701f75f8ee303d94827"}
Sep 16 18:29:27.198309 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:27.198258    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-pssg2" podStartSLOduration=1.496053464 podStartE2EDuration="3.198241996s" totalImagesPullingTime="1.702188532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:27.196049313 +0000 UTC m=+805.060953361" watchObservedRunningTime="2026-09-16 18:29:27.198241996 +0000 UTC m=+805.063146043"
Sep 16 18:29:37.181247 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:37.181215    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-n726w"
Sep 16 18:29:41.560480 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:41.560443    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:41.661310 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:41.661273    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:42.057837 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:42.057787    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:53.889077 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:53.889043    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:54.065200 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.065167    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:54.102190 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.102151    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-stnkc"]
Sep 16 18:29:54.105555 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.105525    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-stnkc"
Sep 16 18:29:54.109699 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.109671    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8flk\" (UniqueName: \"kubernetes.io/projected/42147451-bd6c-4253-92c6-f480a8bba7fa-kube-api-access-q8flk\") pod \"authorino-557f876f69-stnkc\" (UID: \"42147451-bd6c-4253-92c6-f480a8bba7fa\") " pod="kuadrant-system/authorino-557f876f69-stnkc"
Sep 16 18:29:54.115581 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.115554    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-stnkc"]
Sep 16 18:29:54.210502 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.210464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8flk\" (UniqueName: \"kubernetes.io/projected/42147451-bd6c-4253-92c6-f480a8bba7fa-kube-api-access-q8flk\") pod \"authorino-557f876f69-stnkc\" (UID: \"42147451-bd6c-4253-92c6-f480a8bba7fa\") " pod="kuadrant-system/authorino-557f876f69-stnkc"
Sep 16 18:29:54.219477 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.219438    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8flk\" (UniqueName: \"kubernetes.io/projected/42147451-bd6c-4253-92c6-f480a8bba7fa-kube-api-access-q8flk\") pod \"authorino-557f876f69-stnkc\" (UID: \"42147451-bd6c-4253-92c6-f480a8bba7fa\") " pod="kuadrant-system/authorino-557f876f69-stnkc"
Sep 16 18:29:54.232796 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.232746    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-stnkc"]
Sep 16 18:29:54.233040 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.233028    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-stnkc"
Sep 16 18:29:54.265865 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.265829    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9q855"]
Sep 16 18:29:54.269035 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.269009    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9q855"
Sep 16 18:29:54.271643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.271615    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 18:29:54.281761 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.281728    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9q855"]
Sep 16 18:29:54.311070 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.311033    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnvzw\" (UniqueName: \"kubernetes.io/projected/11102180-fddd-4319-9c07-3b3309c143dd-kube-api-access-xnvzw\") pod \"authorino-6d4dfc649f-9q855\" (UID: \"11102180-fddd-4319-9c07-3b3309c143dd\") " pod="kuadrant-system/authorino-6d4dfc649f-9q855"
Sep 16 18:29:54.311239 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.311087    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/11102180-fddd-4319-9c07-3b3309c143dd-tls-cert\") pod \"authorino-6d4dfc649f-9q855\" (UID: \"11102180-fddd-4319-9c07-3b3309c143dd\") " pod="kuadrant-system/authorino-6d4dfc649f-9q855"
Sep 16 18:29:54.335826 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.335781    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9q855"]
Sep 16 18:29:54.336101 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:29:54.336081    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xnvzw tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-9q855" podUID="11102180-fddd-4319-9c07-3b3309c143dd"
Sep 16 18:29:54.368768 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.368727    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-67fff7fbc7-gtktp"]
Sep 16 18:29:54.371268 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.371248    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67fff7fbc7-gtktp"
Sep 16 18:29:54.379224 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.379200    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67fff7fbc7-gtktp"]
Sep 16 18:29:54.411864 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.411816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnlfz\" (UniqueName: \"kubernetes.io/projected/e60cae8f-6756-47f8-aac7-3a52d73bfd1c-kube-api-access-bnlfz\") pod \"authorino-67fff7fbc7-gtktp\" (UID: \"e60cae8f-6756-47f8-aac7-3a52d73bfd1c\") " pod="kuadrant-system/authorino-67fff7fbc7-gtktp"
Sep 16 18:29:54.412067 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.411870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnvzw\" (UniqueName: \"kubernetes.io/projected/11102180-fddd-4319-9c07-3b3309c143dd-kube-api-access-xnvzw\") pod \"authorino-6d4dfc649f-9q855\" (UID: \"11102180-fddd-4319-9c07-3b3309c143dd\") " pod="kuadrant-system/authorino-6d4dfc649f-9q855"
Sep 16 18:29:54.412067 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.411917    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/11102180-fddd-4319-9c07-3b3309c143dd-tls-cert\") pod \"authorino-6d4dfc649f-9q855\" (UID: \"11102180-fddd-4319-9c07-3b3309c143dd\") " pod="kuadrant-system/authorino-6d4dfc649f-9q855"
Sep 16 18:29:54.412067 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.411987    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e60cae8f-6756-47f8-aac7-3a52d73bfd1c-tls-cert\") pod \"authorino-67fff7fbc7-gtktp\" (UID: \"e60cae8f-6756-47f8-aac7-3a52d73bfd1c\") " pod="kuadrant-system/authorino-67fff7fbc7-gtktp"
Sep 16 18:29:54.414498 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.414470    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/11102180-fddd-4319-9c07-3b3309c143dd-tls-cert\") pod \"authorino-6d4dfc649f-9q855\" (UID: \"11102180-fddd-4319-9c07-3b3309c143dd\") " pod="kuadrant-system/authorino-6d4dfc649f-9q855"
Sep 16 18:29:54.420041 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.420016    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnvzw\" (UniqueName: \"kubernetes.io/projected/11102180-fddd-4319-9c07-3b3309c143dd-kube-api-access-xnvzw\") pod \"authorino-6d4dfc649f-9q855\" (UID: \"11102180-fddd-4319-9c07-3b3309c143dd\") " pod="kuadrant-system/authorino-6d4dfc649f-9q855"
Sep 16 18:29:54.508293 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.508264    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-stnkc"]
Sep 16 18:29:54.509350 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:29:54.509328    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42147451_bd6c_4253_92c6_f480a8bba7fa.slice/crio-fe8fa3a4ca8b1d5fb361f9a0e832067731a60e11ac613c47f087e9a8b5c26a36 WatchSource:0}: Error finding container fe8fa3a4ca8b1d5fb361f9a0e832067731a60e11ac613c47f087e9a8b5c26a36: Status 404 returned error can't find the container with id fe8fa3a4ca8b1d5fb361f9a0e832067731a60e11ac613c47f087e9a8b5c26a36
Sep 16 18:29:54.512364 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.512340    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnlfz\" (UniqueName: \"kubernetes.io/projected/e60cae8f-6756-47f8-aac7-3a52d73bfd1c-kube-api-access-bnlfz\") pod \"authorino-67fff7fbc7-gtktp\" (UID: \"e60cae8f-6756-47f8-aac7-3a52d73bfd1c\") " pod="kuadrant-system/authorino-67fff7fbc7-gtktp"
Sep 16 18:29:54.512458 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.512437    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e60cae8f-6756-47f8-aac7-3a52d73bfd1c-tls-cert\") pod \"authorino-67fff7fbc7-gtktp\" (UID: \"e60cae8f-6756-47f8-aac7-3a52d73bfd1c\") " pod="kuadrant-system/authorino-67fff7fbc7-gtktp"
Sep 16 18:29:54.514876 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.514849    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e60cae8f-6756-47f8-aac7-3a52d73bfd1c-tls-cert\") pod \"authorino-67fff7fbc7-gtktp\" (UID: \"e60cae8f-6756-47f8-aac7-3a52d73bfd1c\") " pod="kuadrant-system/authorino-67fff7fbc7-gtktp"
Sep 16 18:29:54.519634 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.519614    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnlfz\" (UniqueName: \"kubernetes.io/projected/e60cae8f-6756-47f8-aac7-3a52d73bfd1c-kube-api-access-bnlfz\") pod \"authorino-67fff7fbc7-gtktp\" (UID: \"e60cae8f-6756-47f8-aac7-3a52d73bfd1c\") " pod="kuadrant-system/authorino-67fff7fbc7-gtktp"
Sep 16 18:29:54.681453 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.681414    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67fff7fbc7-gtktp"
Sep 16 18:29:54.954254 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:54.954225    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67fff7fbc7-gtktp"]
Sep 16 18:29:54.955556 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:29:54.955521    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode60cae8f_6756_47f8_aac7_3a52d73bfd1c.slice/crio-b14d1ba35988c5965a7a5452fefb150a73001072b60c673b725401b191327926 WatchSource:0}: Error finding container b14d1ba35988c5965a7a5452fefb150a73001072b60c673b725401b191327926: Status 404 returned error can't find the container with id b14d1ba35988c5965a7a5452fefb150a73001072b60c673b725401b191327926
Sep 16 18:29:55.331719 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.331624    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67fff7fbc7-gtktp" event={"ID":"e60cae8f-6756-47f8-aac7-3a52d73bfd1c","Type":"ContainerStarted","Data":"b14d1ba35988c5965a7a5452fefb150a73001072b60c673b725401b191327926"}
Sep 16 18:29:55.333334 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.333305    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-stnkc" event={"ID":"42147451-bd6c-4253-92c6-f480a8bba7fa","Type":"ContainerStarted","Data":"303aa5c98b7299dee17a8d5a3ed587d1eede72bb03cac4af8a11aedaf56c89b4"}
Sep 16 18:29:55.333478 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.333338    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9q855"
Sep 16 18:29:55.333478 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.333343    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-stnkc" event={"ID":"42147451-bd6c-4253-92c6-f480a8bba7fa","Type":"ContainerStarted","Data":"fe8fa3a4ca8b1d5fb361f9a0e832067731a60e11ac613c47f087e9a8b5c26a36"}
Sep 16 18:29:55.333478 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.333354    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-stnkc" podUID="42147451-bd6c-4253-92c6-f480a8bba7fa" containerName="authorino" containerID="cri-o://303aa5c98b7299dee17a8d5a3ed587d1eede72bb03cac4af8a11aedaf56c89b4" gracePeriod=30
Sep 16 18:29:55.340398 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.340376    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9q855"
Sep 16 18:29:55.350275 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.350231    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-stnkc" podStartSLOduration=0.763196158 podStartE2EDuration="1.350216193s" totalImagesPullingTime="587.020035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:55.348184546 +0000 UTC m=+833.213088595" watchObservedRunningTime="2026-09-16 18:29:55.350216193 +0000 UTC m=+833.215120571"
Sep 16 18:29:55.418236 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.418201    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/11102180-fddd-4319-9c07-3b3309c143dd-tls-cert\") pod \"11102180-fddd-4319-9c07-3b3309c143dd\" (UID: \"11102180-fddd-4319-9c07-3b3309c143dd\") "
Sep 16 18:29:55.418236 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.418240    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnvzw\" (UniqueName: \"kubernetes.io/projected/11102180-fddd-4319-9c07-3b3309c143dd-kube-api-access-xnvzw\") pod \"11102180-fddd-4319-9c07-3b3309c143dd\" (UID: \"11102180-fddd-4319-9c07-3b3309c143dd\") "
Sep 16 18:29:55.420343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.420316    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11102180-fddd-4319-9c07-3b3309c143dd-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "11102180-fddd-4319-9c07-3b3309c143dd" (UID: "11102180-fddd-4319-9c07-3b3309c143dd"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:29:55.420448 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.420392    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11102180-fddd-4319-9c07-3b3309c143dd-kube-api-access-xnvzw" (OuterVolumeSpecName: "kube-api-access-xnvzw") pod "11102180-fddd-4319-9c07-3b3309c143dd" (UID: "11102180-fddd-4319-9c07-3b3309c143dd"). InnerVolumeSpecName "kube-api-access-xnvzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:29:55.519589 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.519218    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/11102180-fddd-4319-9c07-3b3309c143dd-tls-cert\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:29:55.519589 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.519251    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnvzw\" (UniqueName: \"kubernetes.io/projected/11102180-fddd-4319-9c07-3b3309c143dd-kube-api-access-xnvzw\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:29:55.583628 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.583593    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-stnkc"
Sep 16 18:29:55.720323 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.720288    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8flk\" (UniqueName: \"kubernetes.io/projected/42147451-bd6c-4253-92c6-f480a8bba7fa-kube-api-access-q8flk\") pod \"42147451-bd6c-4253-92c6-f480a8bba7fa\" (UID: \"42147451-bd6c-4253-92c6-f480a8bba7fa\") "
Sep 16 18:29:55.722643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.722581    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42147451-bd6c-4253-92c6-f480a8bba7fa-kube-api-access-q8flk" (OuterVolumeSpecName: "kube-api-access-q8flk") pod "42147451-bd6c-4253-92c6-f480a8bba7fa" (UID: "42147451-bd6c-4253-92c6-f480a8bba7fa"). InnerVolumeSpecName "kube-api-access-q8flk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:29:55.821319 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:55.821278    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8flk\" (UniqueName: \"kubernetes.io/projected/42147451-bd6c-4253-92c6-f480a8bba7fa-kube-api-access-q8flk\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:29:56.343928 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.343889    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67fff7fbc7-gtktp" event={"ID":"e60cae8f-6756-47f8-aac7-3a52d73bfd1c","Type":"ContainerStarted","Data":"c77e7170a39a4af42789d7c2cbded2e623c2d46cae9b8e2ae7b3925e0a92c371"}
Sep 16 18:29:56.345221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.345196    2562 generic.go:356] "Generic (PLEG): container finished" podID="42147451-bd6c-4253-92c6-f480a8bba7fa" containerID="303aa5c98b7299dee17a8d5a3ed587d1eede72bb03cac4af8a11aedaf56c89b4" exitCode=0
Sep 16 18:29:56.345332 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.345228    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-stnkc"
Sep 16 18:29:56.345332 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.345246    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9q855"
Sep 16 18:29:56.345332 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.345273    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-stnkc" event={"ID":"42147451-bd6c-4253-92c6-f480a8bba7fa","Type":"ContainerDied","Data":"303aa5c98b7299dee17a8d5a3ed587d1eede72bb03cac4af8a11aedaf56c89b4"}
Sep 16 18:29:56.345332 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.345313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-stnkc" event={"ID":"42147451-bd6c-4253-92c6-f480a8bba7fa","Type":"ContainerDied","Data":"fe8fa3a4ca8b1d5fb361f9a0e832067731a60e11ac613c47f087e9a8b5c26a36"}
Sep 16 18:29:56.345531 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.345333    2562 scope.go:117] "RemoveContainer" containerID="303aa5c98b7299dee17a8d5a3ed587d1eede72bb03cac4af8a11aedaf56c89b4"
Sep 16 18:29:56.355988 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.355968    2562 scope.go:117] "RemoveContainer" containerID="303aa5c98b7299dee17a8d5a3ed587d1eede72bb03cac4af8a11aedaf56c89b4"
Sep 16 18:29:56.356293 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:29:56.356274    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303aa5c98b7299dee17a8d5a3ed587d1eede72bb03cac4af8a11aedaf56c89b4\": container with ID starting with 303aa5c98b7299dee17a8d5a3ed587d1eede72bb03cac4af8a11aedaf56c89b4 not found: ID does not exist" containerID="303aa5c98b7299dee17a8d5a3ed587d1eede72bb03cac4af8a11aedaf56c89b4"
Sep 16 18:29:56.356363 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.356301    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303aa5c98b7299dee17a8d5a3ed587d1eede72bb03cac4af8a11aedaf56c89b4"} err="failed to get container status \"303aa5c98b7299dee17a8d5a3ed587d1eede72bb03cac4af8a11aedaf56c89b4\": rpc error: code = NotFound desc = could not find container \"303aa5c98b7299dee17a8d5a3ed587d1eede72bb03cac4af8a11aedaf56c89b4\": container with ID starting with 303aa5c98b7299dee17a8d5a3ed587d1eede72bb03cac4af8a11aedaf56c89b4 not found: ID does not exist"
Sep 16 18:29:56.359491 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.359443    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-67fff7fbc7-gtktp" podStartSLOduration=1.734292909 podStartE2EDuration="2.359428015s" totalImagesPullingTime="625.135106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:29:56.357936831 +0000 UTC m=+834.222840880" watchObservedRunningTime="2026-09-16 18:29:56.359428015 +0000 UTC m=+834.224332084"
Sep 16 18:29:56.384346 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.384301    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-pssg2"]
Sep 16 18:29:56.384575 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.384530    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-pssg2" podUID="368bf6ba-c029-400b-830e-1ec83a9b2ef4" containerName="authorino" containerID="cri-o://074eecc3265244f95e4b3afca2317dbe86dd6f313e6aa701f75f8ee303d94827" gracePeriod=30
Sep 16 18:29:56.388904 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.388870    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9q855"]
Sep 16 18:29:56.395720 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.395683    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9q855"]
Sep 16 18:29:56.407197 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.407163    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-stnkc"]
Sep 16 18:29:56.412489 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.412459    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-stnkc"]
Sep 16 18:29:56.552333 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.552295    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11102180-fddd-4319-9c07-3b3309c143dd" path="/var/lib/kubelet/pods/11102180-fddd-4319-9c07-3b3309c143dd/volumes"
Sep 16 18:29:56.552634 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.552619    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42147451-bd6c-4253-92c6-f480a8bba7fa" path="/var/lib/kubelet/pods/42147451-bd6c-4253-92c6-f480a8bba7fa/volumes"
Sep 16 18:29:56.643413 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.643378    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-pssg2"
Sep 16 18:29:56.726872 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.726834    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4jnv\" (UniqueName: \"kubernetes.io/projected/368bf6ba-c029-400b-830e-1ec83a9b2ef4-kube-api-access-q4jnv\") pod \"368bf6ba-c029-400b-830e-1ec83a9b2ef4\" (UID: \"368bf6ba-c029-400b-830e-1ec83a9b2ef4\") "
Sep 16 18:29:56.728930 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.728901    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368bf6ba-c029-400b-830e-1ec83a9b2ef4-kube-api-access-q4jnv" (OuterVolumeSpecName: "kube-api-access-q4jnv") pod "368bf6ba-c029-400b-830e-1ec83a9b2ef4" (UID: "368bf6ba-c029-400b-830e-1ec83a9b2ef4"). InnerVolumeSpecName "kube-api-access-q4jnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:29:56.828019 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:56.827925    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4jnv\" (UniqueName: \"kubernetes.io/projected/368bf6ba-c029-400b-830e-1ec83a9b2ef4-kube-api-access-q4jnv\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:29:57.354058 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:57.354020    2562 generic.go:356] "Generic (PLEG): container finished" podID="368bf6ba-c029-400b-830e-1ec83a9b2ef4" containerID="074eecc3265244f95e4b3afca2317dbe86dd6f313e6aa701f75f8ee303d94827" exitCode=0
Sep 16 18:29:57.354587 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:57.354136    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-pssg2"
Sep 16 18:29:57.354587 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:57.354171    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-pssg2" event={"ID":"368bf6ba-c029-400b-830e-1ec83a9b2ef4","Type":"ContainerDied","Data":"074eecc3265244f95e4b3afca2317dbe86dd6f313e6aa701f75f8ee303d94827"}
Sep 16 18:29:57.354587 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:57.354217    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-pssg2" event={"ID":"368bf6ba-c029-400b-830e-1ec83a9b2ef4","Type":"ContainerDied","Data":"3e81ace2868b67a0cb14051e21b82a7ab6bd1136630f52cb602d9c1aeecbe633"}
Sep 16 18:29:57.354587 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:57.354237    2562 scope.go:117] "RemoveContainer" containerID="074eecc3265244f95e4b3afca2317dbe86dd6f313e6aa701f75f8ee303d94827"
Sep 16 18:29:57.364384 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:57.364351    2562 scope.go:117] "RemoveContainer" containerID="074eecc3265244f95e4b3afca2317dbe86dd6f313e6aa701f75f8ee303d94827"
Sep 16 18:29:57.364760 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:29:57.364735    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074eecc3265244f95e4b3afca2317dbe86dd6f313e6aa701f75f8ee303d94827\": container with ID starting with 074eecc3265244f95e4b3afca2317dbe86dd6f313e6aa701f75f8ee303d94827 not found: ID does not exist" containerID="074eecc3265244f95e4b3afca2317dbe86dd6f313e6aa701f75f8ee303d94827"
Sep 16 18:29:57.364834 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:57.364773    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074eecc3265244f95e4b3afca2317dbe86dd6f313e6aa701f75f8ee303d94827"} err="failed to get container status \"074eecc3265244f95e4b3afca2317dbe86dd6f313e6aa701f75f8ee303d94827\": rpc error: code = NotFound desc = could not find container \"074eecc3265244f95e4b3afca2317dbe86dd6f313e6aa701f75f8ee303d94827\": container with ID starting with 074eecc3265244f95e4b3afca2317dbe86dd6f313e6aa701f75f8ee303d94827 not found: ID does not exist"
Sep 16 18:29:57.376122 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:57.376081    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-pssg2"]
Sep 16 18:29:57.379761 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:57.379727    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-pssg2"]
Sep 16 18:29:58.550530 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:29:58.550490    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="368bf6ba-c029-400b-830e-1ec83a9b2ef4" path="/var/lib/kubelet/pods/368bf6ba-c029-400b-830e-1ec83a9b2ef4/volumes"
Sep 16 18:30:01.025601 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:01.025535    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:01.122065 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:01.122030    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:01.200232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:01.200200    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:01.452217 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:01.452183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:01.550156 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:01.550119    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:05.017805 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:05.017763    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:05.118970 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:05.118933    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:37.556848 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:37.556809    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:38.582298 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:38.582266    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:38.968344 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:38.968304    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:39.072615 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:39.072578    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:39.169924 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:39.169887    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:39.713855 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:39.713823    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:39.824641 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:39.824605    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:30:39.828697 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:39.828669    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:43.396517 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:43.396463    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:49.648693 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:49.648658    2562 generic.go:356] "Generic (PLEG): container finished" podID="2c8ffe4b-cb09-4d16-9740-7eba4c4d404f" containerID="a65376a76981e132d5977015ef9b96ed1b88d576b667c4b1d0be4b640d513b1a" exitCode=0
Sep 16 18:30:49.649148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:49.648737    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv" event={"ID":"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f","Type":"ContainerDied","Data":"a65376a76981e132d5977015ef9b96ed1b88d576b667c4b1d0be4b640d513b1a"}
Sep 16 18:30:49.649148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:49.649135    2562 scope.go:117] "RemoveContainer" containerID="a65376a76981e132d5977015ef9b96ed1b88d576b667c4b1d0be4b640d513b1a"
Sep 16 18:30:49.650146 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:49.650120    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:30:49.650197 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:49.650185    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:30:49.650232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:49.650217    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:30:50.660655 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:50.660607    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv" event={"ID":"2c8ffe4b-cb09-4d16-9740-7eba4c4d404f","Type":"ContainerStarted","Data":"81d360fdb8cc5a2b40002cac11bb7a38e0a34c066200ce024332e2581cae7ca4"}
Sep 16 18:30:50.886346 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:50.886307    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:30:50.887519 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:50.887490    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-2mcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 16 18:30:50.887663 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:50.887554    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv" podUID="2c8ffe4b-cb09-4d16-9740-7eba4c4d404f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 16 18:30:51.098048 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:51.098018    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:51.195712 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:51.195662    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:51.886457 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:51.886416    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-2mcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 16 18:30:51.886904 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:51.886479    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv" podUID="2c8ffe4b-cb09-4d16-9740-7eba4c4d404f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 16 18:30:52.886657 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:52.886612    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-2mcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 16 18:30:52.887063 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:52.886701    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv" podUID="2c8ffe4b-cb09-4d16-9740-7eba4c4d404f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 16 18:30:53.886164 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:53.886120    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-2mcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 16 18:30:53.886360 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:53.886213    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv" podUID="2c8ffe4b-cb09-4d16-9740-7eba4c4d404f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 16 18:30:54.886853 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:54.886815    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-2mcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 16 18:30:54.887359 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:54.886883    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv" podUID="2c8ffe4b-cb09-4d16-9740-7eba4c4d404f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 16 18:30:55.075704 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:55.075651    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:30:55.890613 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:55.890582    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:30:55.891653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:55.891630    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-2mcsv"
Sep 16 18:30:57.947598 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:57.947534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:58.415769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:30:58.415732    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:31:01.005617 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:31:01.005570    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:01.710611 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:31:01.710572    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:31:02.558786 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:31:02.558744    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:31:02.563039 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:31:02.563016    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:31:02.565531 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:31:02.565506    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:31:02.570117 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:31:02.570095    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:31:37.962318 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:31:37.962278    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:39.132159 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:31:39.132118    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:31:44.162921 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:31:44.162880    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:46.661154 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:31:46.661122    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:48.031663 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:31:48.031627    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:32:06.217263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:06.217222    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:32:07.825248 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:07.825207    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:32:25.418612 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:25.418519    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:32:27.225502 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:27.225465    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:32:37.363437 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.363393    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"]
Sep 16 18:32:37.364003 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.363982    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42147451-bd6c-4253-92c6-f480a8bba7fa" containerName="authorino"
Sep 16 18:32:37.364090 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.364006    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="42147451-bd6c-4253-92c6-f480a8bba7fa" containerName="authorino"
Sep 16 18:32:37.364090 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.364041    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="368bf6ba-c029-400b-830e-1ec83a9b2ef4" containerName="authorino"
Sep 16 18:32:37.364090 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.364050    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="368bf6ba-c029-400b-830e-1ec83a9b2ef4" containerName="authorino"
Sep 16 18:32:37.364238 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.364143    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="42147451-bd6c-4253-92c6-f480a8bba7fa" containerName="authorino"
Sep 16 18:32:37.364238 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.364160    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="368bf6ba-c029-400b-830e-1ec83a9b2ef4" containerName="authorino"
Sep 16 18:32:37.366470 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.366446    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.372599 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.372576    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 16 18:32:37.372755 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.372576    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5zsbs\""
Sep 16 18:32:37.372755 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.372578    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 18:32:37.372966 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.372952    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 18:32:37.398483 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.398448    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"]
Sep 16 18:32:37.401945 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.401913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb70fce-2990-44c6-a6aa-205efd099f99-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.402086 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.401947    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb70fce-2990-44c6-a6aa-205efd099f99-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.402086 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.401976    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb70fce-2990-44c6-a6aa-205efd099f99-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.402160 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.402132    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb70fce-2990-44c6-a6aa-205efd099f99-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.402199 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.402184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb70fce-2990-44c6-a6aa-205efd099f99-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.402233 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.402220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpqdq\" (UniqueName: \"kubernetes.io/projected/6bb70fce-2990-44c6-a6aa-205efd099f99-kube-api-access-fpqdq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.431030 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.430997    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:32:37.503054 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.503010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpqdq\" (UniqueName: \"kubernetes.io/projected/6bb70fce-2990-44c6-a6aa-205efd099f99-kube-api-access-fpqdq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.503268 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.503070    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb70fce-2990-44c6-a6aa-205efd099f99-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.503268 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.503087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb70fce-2990-44c6-a6aa-205efd099f99-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.503268 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.503113    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb70fce-2990-44c6-a6aa-205efd099f99-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.503268 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.503164    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb70fce-2990-44c6-a6aa-205efd099f99-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.503268 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.503198    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb70fce-2990-44c6-a6aa-205efd099f99-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.503579 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.503531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb70fce-2990-44c6-a6aa-205efd099f99-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.503651 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.503620    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb70fce-2990-44c6-a6aa-205efd099f99-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.503700 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.503652    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb70fce-2990-44c6-a6aa-205efd099f99-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.505633 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.505605    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb70fce-2990-44c6-a6aa-205efd099f99-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.505907 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.505888    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb70fce-2990-44c6-a6aa-205efd099f99-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.512686 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.512658    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpqdq\" (UniqueName: \"kubernetes.io/projected/6bb70fce-2990-44c6-a6aa-205efd099f99-kube-api-access-fpqdq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv\" (UID: \"6bb70fce-2990-44c6-a6aa-205efd099f99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.676533 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.676431    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:37.966315 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.966286    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"]
Sep 16 18:32:37.968740 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:32:37.968708    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bb70fce_2990_44c6_a6aa_205efd099f99.slice/crio-645119db052b907bf0c4c5a4c888b4c6973cab8be75f1e063d192f031c15e84a WatchSource:0}: Error finding container 645119db052b907bf0c4c5a4c888b4c6973cab8be75f1e063d192f031c15e84a: Status 404 returned error can't find the container with id 645119db052b907bf0c4c5a4c888b4c6973cab8be75f1e063d192f031c15e84a
Sep 16 18:32:37.970460 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:37.970438    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:32:38.283850 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:38.283765    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv" event={"ID":"6bb70fce-2990-44c6-a6aa-205efd099f99","Type":"ContainerStarted","Data":"645119db052b907bf0c4c5a4c888b4c6973cab8be75f1e063d192f031c15e84a"}
Sep 16 18:32:38.667046 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:38.666952    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:32:39.228329 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:39.228243    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:32:39.294763 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:39.294724    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv" event={"ID":"6bb70fce-2990-44c6-a6aa-205efd099f99","Type":"ContainerStarted","Data":"357f57352ff4427f21d614b4ba15947d1eb18a1247aaa3e92e2e625a3fbce978"}
Sep 16 18:32:39.341339 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:39.341285    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv" podStartSLOduration=1.35803103 podStartE2EDuration="2.341266727s" totalImagesPullingTime="983.235697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:32:39.339227169 +0000 UTC m=+997.204131213" watchObservedRunningTime="2026-09-16 18:32:39.341266727 +0000 UTC m=+997.206170775"
Sep 16 18:32:47.677393 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:47.677351    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:47.677850 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:47.677606    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:47.689972 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:47.689940    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:32:48.359161 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:32:48.359131    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv"
Sep 16 18:33:03.171191 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:03.171158    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:33:34.695677 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:34.695629    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:33:49.097056 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:49.097022    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66bc94f9f6-zb4w7"]
Sep 16 18:33:49.100299 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:49.100275    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66bc94f9f6-zb4w7"
Sep 16 18:33:49.111914 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:49.111877    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66bc94f9f6-zb4w7"]
Sep 16 18:33:49.234982 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:49.234939    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/885398f2-92cc-41ef-baa7-15dd7bcb2119-tls-cert\") pod \"authorino-66bc94f9f6-zb4w7\" (UID: \"885398f2-92cc-41ef-baa7-15dd7bcb2119\") " pod="kuadrant-system/authorino-66bc94f9f6-zb4w7"
Sep 16 18:33:49.235175 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:49.235078    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpt9r\" (UniqueName: \"kubernetes.io/projected/885398f2-92cc-41ef-baa7-15dd7bcb2119-kube-api-access-fpt9r\") pod \"authorino-66bc94f9f6-zb4w7\" (UID: \"885398f2-92cc-41ef-baa7-15dd7bcb2119\") " pod="kuadrant-system/authorino-66bc94f9f6-zb4w7"
Sep 16 18:33:49.336330 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:49.336291    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpt9r\" (UniqueName: \"kubernetes.io/projected/885398f2-92cc-41ef-baa7-15dd7bcb2119-kube-api-access-fpt9r\") pod \"authorino-66bc94f9f6-zb4w7\" (UID: \"885398f2-92cc-41ef-baa7-15dd7bcb2119\") " pod="kuadrant-system/authorino-66bc94f9f6-zb4w7"
Sep 16 18:33:49.336330 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:49.336337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/885398f2-92cc-41ef-baa7-15dd7bcb2119-tls-cert\") pod \"authorino-66bc94f9f6-zb4w7\" (UID: \"885398f2-92cc-41ef-baa7-15dd7bcb2119\") " pod="kuadrant-system/authorino-66bc94f9f6-zb4w7"
Sep 16 18:33:49.338846 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:49.338822    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/885398f2-92cc-41ef-baa7-15dd7bcb2119-tls-cert\") pod \"authorino-66bc94f9f6-zb4w7\" (UID: \"885398f2-92cc-41ef-baa7-15dd7bcb2119\") " pod="kuadrant-system/authorino-66bc94f9f6-zb4w7"
Sep 16 18:33:49.353816 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:49.353754    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpt9r\" (UniqueName: \"kubernetes.io/projected/885398f2-92cc-41ef-baa7-15dd7bcb2119-kube-api-access-fpt9r\") pod \"authorino-66bc94f9f6-zb4w7\" (UID: \"885398f2-92cc-41ef-baa7-15dd7bcb2119\") " pod="kuadrant-system/authorino-66bc94f9f6-zb4w7"
Sep 16 18:33:49.410301 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:49.410265    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66bc94f9f6-zb4w7"
Sep 16 18:33:49.695493 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:49.695458    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66bc94f9f6-zb4w7"]
Sep 16 18:33:49.697115 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:33:49.697086    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod885398f2_92cc_41ef_baa7_15dd7bcb2119.slice/crio-3f2f72f54d22475f7a77742f65cb99511e9b2578cb553faa5a722b5260d0ac3d WatchSource:0}: Error finding container 3f2f72f54d22475f7a77742f65cb99511e9b2578cb553faa5a722b5260d0ac3d: Status 404 returned error can't find the container with id 3f2f72f54d22475f7a77742f65cb99511e9b2578cb553faa5a722b5260d0ac3d
Sep 16 18:33:49.705280 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:49.705253    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66bc94f9f6-zb4w7" event={"ID":"885398f2-92cc-41ef-baa7-15dd7bcb2119","Type":"ContainerStarted","Data":"3f2f72f54d22475f7a77742f65cb99511e9b2578cb553faa5a722b5260d0ac3d"}
Sep 16 18:33:50.713003 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:50.712965    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66bc94f9f6-zb4w7" event={"ID":"885398f2-92cc-41ef-baa7-15dd7bcb2119","Type":"ContainerStarted","Data":"662ad60bebe506c8779574329e5054cec3ae9b5b37e61426c82bd4e4e255056f"}
Sep 16 18:33:50.744517 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:50.744458    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-66bc94f9f6-zb4w7" podStartSLOduration=0.892678738 podStartE2EDuration="1.744441902s" totalImagesPullingTime="851.763164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:33:50.74275825 +0000 UTC m=+1068.607662299" watchObservedRunningTime="2026-09-16 18:33:50.744441902 +0000 UTC m=+1068.609345950"
Sep 16 18:33:50.780175 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:50.780129    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-67fff7fbc7-gtktp"]
Sep 16 18:33:50.780392 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:50.780356    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-67fff7fbc7-gtktp" podUID="e60cae8f-6756-47f8-aac7-3a52d73bfd1c" containerName="authorino" containerID="cri-o://c77e7170a39a4af42789d7c2cbded2e623c2d46cae9b8e2ae7b3925e0a92c371" gracePeriod=30
Sep 16 18:33:51.056303 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.056227    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67fff7fbc7-gtktp"
Sep 16 18:33:51.152288 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.152255    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnlfz\" (UniqueName: \"kubernetes.io/projected/e60cae8f-6756-47f8-aac7-3a52d73bfd1c-kube-api-access-bnlfz\") pod \"e60cae8f-6756-47f8-aac7-3a52d73bfd1c\" (UID: \"e60cae8f-6756-47f8-aac7-3a52d73bfd1c\") "
Sep 16 18:33:51.152457 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.152369    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e60cae8f-6756-47f8-aac7-3a52d73bfd1c-tls-cert\") pod \"e60cae8f-6756-47f8-aac7-3a52d73bfd1c\" (UID: \"e60cae8f-6756-47f8-aac7-3a52d73bfd1c\") "
Sep 16 18:33:51.154606 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.154529    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60cae8f-6756-47f8-aac7-3a52d73bfd1c-kube-api-access-bnlfz" (OuterVolumeSpecName: "kube-api-access-bnlfz") pod "e60cae8f-6756-47f8-aac7-3a52d73bfd1c" (UID: "e60cae8f-6756-47f8-aac7-3a52d73bfd1c"). InnerVolumeSpecName "kube-api-access-bnlfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:33:51.163166 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.163136    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e60cae8f-6756-47f8-aac7-3a52d73bfd1c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e60cae8f-6756-47f8-aac7-3a52d73bfd1c" (UID: "e60cae8f-6756-47f8-aac7-3a52d73bfd1c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:33:51.253095 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.253052    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e60cae8f-6756-47f8-aac7-3a52d73bfd1c-tls-cert\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:33:51.253095 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.253095    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnlfz\" (UniqueName: \"kubernetes.io/projected/e60cae8f-6756-47f8-aac7-3a52d73bfd1c-kube-api-access-bnlfz\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:33:51.368832 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.368746    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:33:51.506239 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.506197    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:33:51.720819 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.720785    2562 generic.go:356] "Generic (PLEG): container finished" podID="e60cae8f-6756-47f8-aac7-3a52d73bfd1c" containerID="c77e7170a39a4af42789d7c2cbded2e623c2d46cae9b8e2ae7b3925e0a92c371" exitCode=0
Sep 16 18:33:51.721298 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.720827    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67fff7fbc7-gtktp"
Sep 16 18:33:51.721298 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.720878    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67fff7fbc7-gtktp" event={"ID":"e60cae8f-6756-47f8-aac7-3a52d73bfd1c","Type":"ContainerDied","Data":"c77e7170a39a4af42789d7c2cbded2e623c2d46cae9b8e2ae7b3925e0a92c371"}
Sep 16 18:33:51.721298 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.720927    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67fff7fbc7-gtktp" event={"ID":"e60cae8f-6756-47f8-aac7-3a52d73bfd1c","Type":"ContainerDied","Data":"b14d1ba35988c5965a7a5452fefb150a73001072b60c673b725401b191327926"}
Sep 16 18:33:51.721298 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.720953    2562 scope.go:117] "RemoveContainer" containerID="c77e7170a39a4af42789d7c2cbded2e623c2d46cae9b8e2ae7b3925e0a92c371"
Sep 16 18:33:51.731407 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.731389    2562 scope.go:117] "RemoveContainer" containerID="c77e7170a39a4af42789d7c2cbded2e623c2d46cae9b8e2ae7b3925e0a92c371"
Sep 16 18:33:51.731763 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:33:51.731740    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c77e7170a39a4af42789d7c2cbded2e623c2d46cae9b8e2ae7b3925e0a92c371\": container with ID starting with c77e7170a39a4af42789d7c2cbded2e623c2d46cae9b8e2ae7b3925e0a92c371 not found: ID does not exist" containerID="c77e7170a39a4af42789d7c2cbded2e623c2d46cae9b8e2ae7b3925e0a92c371"
Sep 16 18:33:51.731851 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.731770    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c77e7170a39a4af42789d7c2cbded2e623c2d46cae9b8e2ae7b3925e0a92c371"} err="failed to get container status \"c77e7170a39a4af42789d7c2cbded2e623c2d46cae9b8e2ae7b3925e0a92c371\": rpc error: code = NotFound desc = could not find container \"c77e7170a39a4af42789d7c2cbded2e623c2d46cae9b8e2ae7b3925e0a92c371\": container with ID starting with c77e7170a39a4af42789d7c2cbded2e623c2d46cae9b8e2ae7b3925e0a92c371 not found: ID does not exist"
Sep 16 18:33:51.754778 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.754739    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-67fff7fbc7-gtktp"]
Sep 16 18:33:51.757745 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:51.757715    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-67fff7fbc7-gtktp"]
Sep 16 18:33:52.482628 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:52.482585    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:33:52.552345 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:52.552313    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e60cae8f-6756-47f8-aac7-3a52d73bfd1c" path="/var/lib/kubelet/pods/e60cae8f-6756-47f8-aac7-3a52d73bfd1c/volumes"
Sep 16 18:33:52.572031 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:33:52.572002    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:11.491294 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.491256    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:11.630595 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.630524    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:11.854344 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.854267    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"]
Sep 16 18:34:11.854685 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.854672    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e60cae8f-6756-47f8-aac7-3a52d73bfd1c" containerName="authorino"
Sep 16 18:34:11.854733 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.854686    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e60cae8f-6756-47f8-aac7-3a52d73bfd1c" containerName="authorino"
Sep 16 18:34:11.854768 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.854748    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e60cae8f-6756-47f8-aac7-3a52d73bfd1c" containerName="authorino"
Sep 16 18:34:11.859290 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.859268    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:11.862710 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.862684    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-909a40-openshift-default-dockercfg-vcw7c\""
Sep 16 18:34:11.902888 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.902850    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:11.903162 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.903141    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"]
Sep 16 18:34:11.940768 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.940734    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbf4w\" (UniqueName: \"kubernetes.io/projected/81ae555d-de1f-4a84-9788-84ffc4816fd9-kube-api-access-bbf4w\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:11.940988 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.940802    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-data\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:11.940988 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.940833    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-credential-socket\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:11.940988 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.940901    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-envoy\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:11.940988 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.940925    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-podinfo\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:11.940988 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.940964    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ae555d-de1f-4a84-9788-84ffc4816fd9-istiod-ca-cert\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:11.941230 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.941013    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-workload-socket\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:11.941230 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.941040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-token\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:11.941230 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:11.941088    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-workload-certs\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.012523 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.012485    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"]
Sep 16 18:34:12.016394 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.016368    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.027141 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.027113    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4ab744-openshift-default-dockercfg-f6g2l\""
Sep 16 18:34:12.042002 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.041952    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-credential-socket\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.042002 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.041994    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-data\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.042242 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042053    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ae555d-de1f-4a84-9788-84ffc4816fd9-istiod-ca-cert\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.042242 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042085    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-podinfo\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.042242 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042102    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istiod-ca-cert\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.042242 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-workload-socket\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.042242 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-token\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.042562 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042329    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-workload-socket\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.042562 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-workload-certs\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.042562 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042504    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-workload-certs\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.042562 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042537    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbf4w\" (UniqueName: \"kubernetes.io/projected/81ae555d-de1f-4a84-9788-84ffc4816fd9-kube-api-access-bbf4w\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.042830 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042586    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-envoy\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.042830 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042632    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-token\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.042830 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042648    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-workload-socket\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.042830 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-data\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.042830 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042773    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-credential-socket\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.042830 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042798    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cspk\" (UniqueName: \"kubernetes.io/projected/04c9e55b-1b9f-4eda-866b-a9060c4eb871-kube-api-access-5cspk\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.043145 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-workload-certs\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.043145 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042867    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-envoy\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.043145 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042894    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-podinfo\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.043145 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.043077    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-credential-socket\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.043145 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.042869    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ae555d-de1f-4a84-9788-84ffc4816fd9-istiod-ca-cert\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.043366 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.043309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-data\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.045223 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.045200    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-envoy\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.045527 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.045504    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-podinfo\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.057780 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.057741    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"]
Sep 16 18:34:12.066122 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.066088    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:12.067888 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.067854    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-token\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.068100 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.068082    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbf4w\" (UniqueName: \"kubernetes.io/projected/81ae555d-de1f-4a84-9788-84ffc4816fd9-kube-api-access-bbf4w\") pod \"e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") " pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.140947 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.140858    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"]
Sep 16 18:34:12.144361 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.144330    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-credential-socket\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.144504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.144366    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-data\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.144504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.144389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-podinfo\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.144504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.144407    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istiod-ca-cert\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.144504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.144437    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-workload-socket\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.144504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.144486    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-workload-certs\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.144804 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.144515    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-envoy\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.144804 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.144645    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-token\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.144804 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.144708    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cspk\" (UniqueName: \"kubernetes.io/projected/04c9e55b-1b9f-4eda-866b-a9060c4eb871-kube-api-access-5cspk\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.144968 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.144827    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-data\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.144968 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.144868    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-credential-socket\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.144968 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.144911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-workload-socket\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.145162 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.145139    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-workload-certs\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.145348 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.145325    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istiod-ca-cert\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.145726 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.145709    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.146823 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.146804    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-envoy\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.147295 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.147273    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-podinfo\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.148902 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.148879    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d8a600-openshift-default-dockercfg-rxmld\""
Sep 16 18:34:12.170279 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.170246    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:12.180219 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.180189    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-token\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.180642 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.180609    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cspk\" (UniqueName: \"kubernetes.io/projected/04c9e55b-1b9f-4eda-866b-a9060c4eb871-kube-api-access-5cspk\") pod \"e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") " pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.181606 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.181577    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"]
Sep 16 18:34:12.184516 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.184488    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:12.246573 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.245519    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-workload-certs\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.246573 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.245631    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-envoy\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.246573 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.245664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec6d40bd-176d-4785-833d-d894891fbe1f-istiod-ca-cert\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.246573 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.245695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-workload-socket\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.246573 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.245784    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-podinfo\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.246573 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.245836    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-data\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.246573 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.245867    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-credential-socket\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.246573 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.245893    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rdzd\" (UniqueName: \"kubernetes.io/projected/ec6d40bd-176d-4785-833d-d894891fbe1f-kube-api-access-2rdzd\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.246573 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.245941    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-token\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.327663 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.327615    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:12.344683 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.344648    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:12.346888 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.346862    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-podinfo\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.347019 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.346933    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-data\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.347019 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.346966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-credential-socket\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.347019 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.346993    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rdzd\" (UniqueName: \"kubernetes.io/projected/ec6d40bd-176d-4785-833d-d894891fbe1f-kube-api-access-2rdzd\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.347191 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.347044    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-token\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.347191 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.347100    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-workload-certs\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.347191 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.347134    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-envoy\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.347191 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.347160    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec6d40bd-176d-4785-833d-d894891fbe1f-istiod-ca-cert\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.347191 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.347189    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-workload-socket\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.347455 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.347362    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-data\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.347455 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.347381    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-credential-socket\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.347661 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.347641    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-workload-certs\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.347740 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.347716    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-workload-socket\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.347978 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.347955    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec6d40bd-176d-4785-833d-d894891fbe1f-istiod-ca-cert\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.349769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.349741    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-podinfo\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.350347 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.350322    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-envoy\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.371172 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.371136    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rdzd\" (UniqueName: \"kubernetes.io/projected/ec6d40bd-176d-4785-833d-d894891fbe1f-kube-api-access-2rdzd\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.392374 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.392314    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-token\") pod \"e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") " pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.466306 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.466265    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:12.477478 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.477447    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"]
Sep 16 18:34:12.479250 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:34:12.479222    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ae555d_de1f_4a84_9788_84ffc4816fd9.slice/crio-6a0e7bc76e9437206aec4f3d1be74e86aae6a63aa7685524dadb54032afdd3b2 WatchSource:0}: Error finding container 6a0e7bc76e9437206aec4f3d1be74e86aae6a63aa7685524dadb54032afdd3b2: Status 404 returned error can't find the container with id 6a0e7bc76e9437206aec4f3d1be74e86aae6a63aa7685524dadb54032afdd3b2
Sep 16 18:34:12.481735 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.481636    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:12.481735 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.481710    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:12.481810 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.481746    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:12.512070 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.512026    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:12.638776 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.638657    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"]
Sep 16 18:34:12.640003 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.639962    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:12.640126 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.640052    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:12.640126 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.640100    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:12.772823 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.772791    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"]
Sep 16 18:34:12.775439 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:34:12.775384    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6d40bd_176d_4785_833d_d894891fbe1f.slice/crio-9a5a559d7fe9cecbe7154ee82ceb09c15c9d58f755f05e5441d16958dce2681e WatchSource:0}: Error finding container 9a5a559d7fe9cecbe7154ee82ceb09c15c9d58f755f05e5441d16958dce2681e: Status 404 returned error can't find the container with id 9a5a559d7fe9cecbe7154ee82ceb09c15c9d58f755f05e5441d16958dce2681e
Sep 16 18:34:12.778450 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.778408    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:12.778578 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.778493    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:12.778578 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.778564    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:12.847291 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.847162    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27" event={"ID":"81ae555d-de1f-4a84-9788-84ffc4816fd9","Type":"ContainerStarted","Data":"8b3410298278bdf3ecf5deb02d715dde4dd3a69993c80d3a4e2fdc7602f51575"}
Sep 16 18:34:12.847291 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.847242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27" event={"ID":"81ae555d-de1f-4a84-9788-84ffc4816fd9","Type":"ContainerStarted","Data":"6a0e7bc76e9437206aec4f3d1be74e86aae6a63aa7685524dadb54032afdd3b2"}
Sep 16 18:34:12.852903 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.852159    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np" event={"ID":"04c9e55b-1b9f-4eda-866b-a9060c4eb871","Type":"ContainerStarted","Data":"09e927d77dac8671838d16aaeae9ebe7ffc62964fd0cba6a6ddb0cbe0fea53cf"}
Sep 16 18:34:12.852903 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.852228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np" event={"ID":"04c9e55b-1b9f-4eda-866b-a9060c4eb871","Type":"ContainerStarted","Data":"00391a7e3c10bc13e76dbb390c84565668747632d68cee5f32ee43f70f6a7321"}
Sep 16 18:34:12.855302 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.855211    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh" event={"ID":"ec6d40bd-176d-4785-833d-d894891fbe1f","Type":"ContainerStarted","Data":"9a5a559d7fe9cecbe7154ee82ceb09c15c9d58f755f05e5441d16958dce2681e"}
Sep 16 18:34:12.887017 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.886932    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27" podStartSLOduration=1.886904584 podStartE2EDuration="1.886904584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:34:12.88226584 +0000 UTC m=+1090.747169892" watchObservedRunningTime="2026-09-16 18:34:12.886904584 +0000 UTC m=+1090.751808634"
Sep 16 18:34:12.891134 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.891102    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:12.921215 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:12.921087    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np" podStartSLOduration=1.92106165 podStartE2EDuration="1.92106165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:34:12.91691797 +0000 UTC m=+1090.781822019" watchObservedRunningTime="2026-09-16 18:34:12.92106165 +0000 UTC m=+1090.785965699"
Sep 16 18:34:13.014053 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.014018    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:13.171346 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.171254    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:13.176933 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.176904    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:13.328245 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.328202    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:13.333376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.333347    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:13.378421 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.378383    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:13.575024 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.574989    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7885-095348"]
Sep 16 18:34:13.578538 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.578514    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-095348"
Sep 16 18:34:13.583025 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.582996    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fd5lk\""
Sep 16 18:34:13.599294 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.599261    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7885-095348"]
Sep 16 18:34:13.663263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.663220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxk2w\" (UniqueName: \"kubernetes.io/projected/b7e80c68-1468-4701-b330-df296b779214-kube-api-access-fxk2w\") pod \"test-curl-7885-095348\" (UID: \"b7e80c68-1468-4701-b330-df296b779214\") " pod="opendatahub/test-curl-7885-095348"
Sep 16 18:34:13.764078 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.764037    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxk2w\" (UniqueName: \"kubernetes.io/projected/b7e80c68-1468-4701-b330-df296b779214-kube-api-access-fxk2w\") pod \"test-curl-7885-095348\" (UID: \"b7e80c68-1468-4701-b330-df296b779214\") " pod="opendatahub/test-curl-7885-095348"
Sep 16 18:34:13.776683 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.776651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxk2w\" (UniqueName: \"kubernetes.io/projected/b7e80c68-1468-4701-b330-df296b779214-kube-api-access-fxk2w\") pod \"test-curl-7885-095348\" (UID: \"b7e80c68-1468-4701-b330-df296b779214\") " pod="opendatahub/test-curl-7885-095348"
Sep 16 18:34:13.870420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.870320    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh" event={"ID":"ec6d40bd-176d-4785-833d-d894891fbe1f","Type":"ContainerStarted","Data":"33f49ca09ec6ea51a537233df1753b70ec6baf0ef9ac1307545bed63bf7eff89"}
Sep 16 18:34:13.870761 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.870731    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:13.870862 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.870787    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:13.871526 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.871502    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:13.871661 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.871581    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:13.889143 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.889109    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-095348"
Sep 16 18:34:13.912274 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:13.912207    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh" podStartSLOduration=1.912187366 podStartE2EDuration="1.912187366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:34:13.910170092 +0000 UTC m=+1091.775074140" watchObservedRunningTime="2026-09-16 18:34:13.912187366 +0000 UTC m=+1091.777091414"
Sep 16 18:34:14.223622 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:14.223595    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7885-095348"]
Sep 16 18:34:14.225804 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:34:14.225775    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7e80c68_1468_4701_b330_df296b779214.slice/crio-23ee494477b9e5b8b766feea422904709477b18ed5a90250c5a9a9800b8fb9dc WatchSource:0}: Error finding container 23ee494477b9e5b8b766feea422904709477b18ed5a90250c5a9a9800b8fb9dc: Status 404 returned error can't find the container with id 23ee494477b9e5b8b766feea422904709477b18ed5a90250c5a9a9800b8fb9dc
Sep 16 18:34:14.467156 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:14.467119    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:14.472837 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:14.472754    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:14.877797 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:14.877694    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7885-095348" event={"ID":"b7e80c68-1468-4701-b330-df296b779214","Type":"ContainerStarted","Data":"23ee494477b9e5b8b766feea422904709477b18ed5a90250c5a9a9800b8fb9dc"}
Sep 16 18:34:14.878167 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:14.877974    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:14.879057 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:14.879040    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:15.022420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:15.022386    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:34:15.194181 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:15.194081    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:15.436935 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:15.436901    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:15.702028 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:15.701997    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:17.357964 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:17.357921    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:17.563483 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:17.563441    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:18.278000 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:18.277968    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:18.477141 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:18.477109    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:18.725648 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:18.725604    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:18.905745 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:18.905653    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7885-095348" event={"ID":"b7e80c68-1468-4701-b330-df296b779214","Type":"ContainerStarted","Data":"b488343d6a26a856c03ee8a6a137aa4fac09777c084e4fcf395d6a1a7bf8d5ae"}
Sep 16 18:34:18.933184 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:18.933126    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7885-095348" podStartSLOduration=1.541439397 podStartE2EDuration="5.933111323s" totalImagesPullingTime="4.391671926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:34:18.929821949 +0000 UTC m=+1096.794725997" watchObservedRunningTime="2026-09-16 18:34:18.933111323 +0000 UTC m=+1096.798015371"
Sep 16 18:34:19.742166 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:19.742070    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:19.889902 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:19.889865    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7885-095348"]
Sep 16 18:34:19.908516 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:19.908482    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7885-095348"]
Sep 16 18:34:19.913835 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:34:19.913802    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7885-095348" container="test-curl-7885-095348"
Sep 16 18:34:19.915395 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:19.915367    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:20.829868 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:20.829830    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:20.919729 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:20.919687    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7885-095348" podUID="b7e80c68-1468-4701-b330-df296b779214" containerName="test-curl-7885-095348" containerID="cri-o://b488343d6a26a856c03ee8a6a137aa4fac09777c084e4fcf395d6a1a7bf8d5ae" gracePeriod=2
Sep 16 18:34:20.963764 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:20.963731    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:21.215333 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:21.215298    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:22.842341 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:22.842309    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:23.096743 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:23.096669    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:23.120345 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:23.120317    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-095348"
Sep 16 18:34:23.161729 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:23.161688    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxk2w\" (UniqueName: \"kubernetes.io/projected/b7e80c68-1468-4701-b330-df296b779214-kube-api-access-fxk2w\") pod \"b7e80c68-1468-4701-b330-df296b779214\" (UID: \"b7e80c68-1468-4701-b330-df296b779214\") "
Sep 16 18:34:23.164205 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:23.164134    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e80c68-1468-4701-b330-df296b779214-kube-api-access-fxk2w" (OuterVolumeSpecName: "kube-api-access-fxk2w") pod "b7e80c68-1468-4701-b330-df296b779214" (UID: "b7e80c68-1468-4701-b330-df296b779214"). InnerVolumeSpecName "kube-api-access-fxk2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:34:23.262753 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:23.262715    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxk2w\" (UniqueName: \"kubernetes.io/projected/b7e80c68-1468-4701-b330-df296b779214-kube-api-access-fxk2w\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:34:23.942766 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:23.942726    2562 generic.go:356] "Generic (PLEG): container finished" podID="b7e80c68-1468-4701-b330-df296b779214" containerID="b488343d6a26a856c03ee8a6a137aa4fac09777c084e4fcf395d6a1a7bf8d5ae" exitCode=137
Sep 16 18:34:23.943212 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:23.942772    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7885-095348"
Sep 16 18:34:23.943212 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:23.942777    2562 scope.go:117] "RemoveContainer" containerID="b488343d6a26a856c03ee8a6a137aa4fac09777c084e4fcf395d6a1a7bf8d5ae"
Sep 16 18:34:23.956519 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:23.956492    2562 scope.go:117] "RemoveContainer" containerID="b488343d6a26a856c03ee8a6a137aa4fac09777c084e4fcf395d6a1a7bf8d5ae"
Sep 16 18:34:23.956964 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:34:23.956929    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b488343d6a26a856c03ee8a6a137aa4fac09777c084e4fcf395d6a1a7bf8d5ae\": container with ID starting with b488343d6a26a856c03ee8a6a137aa4fac09777c084e4fcf395d6a1a7bf8d5ae not found: ID does not exist" containerID="b488343d6a26a856c03ee8a6a137aa4fac09777c084e4fcf395d6a1a7bf8d5ae"
Sep 16 18:34:23.957069 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:23.956978    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b488343d6a26a856c03ee8a6a137aa4fac09777c084e4fcf395d6a1a7bf8d5ae"} err="failed to get container status \"b488343d6a26a856c03ee8a6a137aa4fac09777c084e4fcf395d6a1a7bf8d5ae\": rpc error: code = NotFound desc = could not find container \"b488343d6a26a856c03ee8a6a137aa4fac09777c084e4fcf395d6a1a7bf8d5ae\": container with ID starting with b488343d6a26a856c03ee8a6a137aa4fac09777c084e4fcf395d6a1a7bf8d5ae not found: ID does not exist"
Sep 16 18:34:23.965288 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:34:23.965242    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7885-095348\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" podUID="b7e80c68-1468-4701-b330-df296b779214" pod="opendatahub/test-curl-7885-095348"
Sep 16 18:34:24.537615 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:24.537570    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:24.575467 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:24.575427    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7e80c68-1468-4701-b330-df296b779214" path="/var/lib/kubelet/pods/b7e80c68-1468-4701-b330-df296b779214/volumes"
Sep 16 18:34:24.756300 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:24.756255    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:24.917256 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:24.917166    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:25.268242 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:25.268203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:25.991965 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:25.991925    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"]
Sep 16 18:34:25.992416 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:25.992397    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7e80c68-1468-4701-b330-df296b779214" containerName="test-curl-7885-095348"
Sep 16 18:34:25.992526 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:25.992419    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7e80c68-1468-4701-b330-df296b779214" containerName="test-curl-7885-095348"
Sep 16 18:34:25.992605 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:25.992595    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7e80c68-1468-4701-b330-df296b779214" containerName="test-curl-7885-095348"
Sep 16 18:34:25.997280 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:25.997253    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.007604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.007570    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-8b875d-openshift-default-dockercfg-mb22d\""
Sep 16 18:34:26.032466 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.032436    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"]
Sep 16 18:34:26.091011 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.090972    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-credential-socket\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.091177 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.091025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istiod-ca-cert\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.091177 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.091067    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-workload-socket\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.091177 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.091091    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-data\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.091177 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.091108    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-workload-certs\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.091177 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.091122    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q9f4\" (UniqueName: \"kubernetes.io/projected/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-kube-api-access-9q9f4\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.091447 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.091243    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-envoy\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.091447 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.091317    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-podinfo\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.091447 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.091351    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-token\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.122298 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.122265    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:26.192442 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.192409    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istiod-ca-cert\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.192442 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.192448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-workload-socket\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.192694 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.192470    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-data\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.192694 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.192498    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-workload-certs\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.192694 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.192522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q9f4\" (UniqueName: \"kubernetes.io/projected/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-kube-api-access-9q9f4\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.192694 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.192611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-envoy\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.192694 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.192666    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-podinfo\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.192899 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.192696    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-token\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.192899 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.192769    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-credential-socket\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.193182 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.193153    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-credential-socket\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.193274 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.193257    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istiod-ca-cert\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.193412 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.193364    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-workload-certs\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.193778 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.193754    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-workload-socket\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.193875 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.193855    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-data\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.195461 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.195438    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-envoy\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.195765 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.195746    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-podinfo\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.282917 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.282833    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-token\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.284991 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.284966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q9f4\" (UniqueName: \"kubernetes.io/projected/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-kube-api-access-9q9f4\") pod \"e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") " pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.311157 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.311113    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:26.534090 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.534007    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:26.684974 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.684925    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"]
Sep 16 18:34:26.686248 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:34:26.686222    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e520ad7_28d2_4a42_8bf3_30e3159a53f9.slice/crio-cbd5c748751ae42e399e3f59f7f01e3c936c21e7b7de124c0b54794213c3c003 WatchSource:0}: Error finding container cbd5c748751ae42e399e3f59f7f01e3c936c21e7b7de124c0b54794213c3c003: Status 404 returned error can't find the container with id cbd5c748751ae42e399e3f59f7f01e3c936c21e7b7de124c0b54794213c3c003
Sep 16 18:34:26.688748 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.688710    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:26.688842 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.688781    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:26.688842 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.688809    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:26.784172 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.784070    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:26.983788 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.983745    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp" event={"ID":"2e520ad7-28d2-4a42-8bf3-30e3159a53f9","Type":"ContainerStarted","Data":"85858c37c4e91192fb10fe9ecc8d21b23675b2840d8ae50a2a9b1db2818c61c5"}
Sep 16 18:34:26.984012 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:26.983814    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp" event={"ID":"2e520ad7-28d2-4a42-8bf3-30e3159a53f9","Type":"ContainerStarted","Data":"cbd5c748751ae42e399e3f59f7f01e3c936c21e7b7de124c0b54794213c3c003"}
Sep 16 18:34:27.044979 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:27.044870    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp" podStartSLOduration=2.044851554 podStartE2EDuration="2.044851554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:34:27.043361364 +0000 UTC m=+1104.908265422" watchObservedRunningTime="2026-09-16 18:34:27.044851554 +0000 UTC m=+1104.909755602"
Sep 16 18:34:27.056889 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:27.056848    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:27.312437 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:27.312335    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:27.317842 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:27.317813    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:27.990279 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:27.990243    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:27.991197 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:27.991177    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:34:29.119773 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:29.119738    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:30.034502 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:30.034467    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:30.392632 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:30.392527    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:32.341477 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:32.341443    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:32.556269 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:32.556237    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:32.730022 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:32.729987    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:33.023338 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.023245    2562 generic.go:356] "Generic (PLEG): container finished" podID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerID="09e927d77dac8671838d16aaeae9ebe7ffc62964fd0cba6a6ddb0cbe0fea53cf" exitCode=0
Sep 16 18:34:33.023338 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.023322    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np" event={"ID":"04c9e55b-1b9f-4eda-866b-a9060c4eb871","Type":"ContainerDied","Data":"09e927d77dac8671838d16aaeae9ebe7ffc62964fd0cba6a6ddb0cbe0fea53cf"}
Sep 16 18:34:33.023797 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.023779    2562 scope.go:117] "RemoveContainer" containerID="09e927d77dac8671838d16aaeae9ebe7ffc62964fd0cba6a6ddb0cbe0fea53cf"
Sep 16 18:34:33.025088 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.025061    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:33.025173 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.025131    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:33.025173 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.025162    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:33.239870 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.239835    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"]
Sep 16 18:34:33.242719 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.242699    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.256239 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.256210    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-119aafd2-openshift-default-dockercfg-4dx2s\""
Sep 16 18:34:33.279037 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.278940    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:33.279259 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.279238    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"]
Sep 16 18:34:33.364221 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.364180    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-credential-socket\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.364742 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.364360    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-workload-certs\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.364742 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.364416    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-workload-socket\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.364742 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.364474    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-data\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.364742 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.364502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-envoy\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.364742 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.364664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-podinfo\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.364742 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.364732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-token\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.365025 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.364773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhq4f\" (UniqueName: \"kubernetes.io/projected/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-kube-api-access-dhq4f\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.365025 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.364849    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istiod-ca-cert\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.465921 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.465885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istiod-ca-cert\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.466127 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.465932    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-credential-socket\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.466127 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.466001    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-workload-certs\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.466127 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.466021    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-workload-socket\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.466127 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.466038    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-data\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.466127 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.466055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-envoy\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.466127 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.466120    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-podinfo\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.466436 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.466237    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-token\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.466497 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.466472    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-workload-socket\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.466572 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.466428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhq4f\" (UniqueName: \"kubernetes.io/projected/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-kube-api-access-dhq4f\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.466572 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.466513    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-workload-certs\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.466682 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.466623    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-credential-socket\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.466771 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.466749    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istiod-ca-cert\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.466979 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.466953    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-data\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.469138 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.469114    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-envoy\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.469264 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.469244    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-podinfo\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.513949 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.513911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhq4f\" (UniqueName: \"kubernetes.io/projected/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-kube-api-access-dhq4f\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.557462 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.557391    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-token\") pod \"e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") " pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.610059 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.610020    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:33.856677 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.856574    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:33.927281 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:33.927193    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:34.031442 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:34.031401    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np" event={"ID":"04c9e55b-1b9f-4eda-866b-a9060c4eb871","Type":"ContainerStarted","Data":"8a156642174c1c1c234eb8b39c9f200af3ab9ab89df86e69c6cf2aaff298efee"}
Sep 16 18:34:34.198441 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:34.198355    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:34.297011 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:34.296977    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"]
Sep 16 18:34:34.298506 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:34:34.298484    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee0c731_debf_4fc9_afc2_8f8a54d279ca.slice/crio-ba74884f3bc880eb9de3e94336479089e5e2b244d84d945510cae450cfc074f7 WatchSource:0}: Error finding container ba74884f3bc880eb9de3e94336479089e5e2b244d84d945510cae450cfc074f7: Status 404 returned error can't find the container with id ba74884f3bc880eb9de3e94336479089e5e2b244d84d945510cae450cfc074f7
Sep 16 18:34:34.300913 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:34.300880    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:34.300989 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:34.300948    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:34.300989 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:34.300972    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:34.327981 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:34.327955    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:34.328156 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:34.328134    2562 patch_prober.go:28] interesting pod/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 18:34:34.328207 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:34.328183    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np" podUID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 18:34:34.656168 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:34.656132    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:35.042611 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:35.042572    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" event={"ID":"7ee0c731-debf-4fc9-afc2-8f8a54d279ca","Type":"ContainerStarted","Data":"235b0d4e0a10ef343a35f9c032acd9d4bf0b33c08885cf4b6cae408bf6d70e2b"}
Sep 16 18:34:35.042826 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:35.042638    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" event={"ID":"7ee0c731-debf-4fc9-afc2-8f8a54d279ca","Type":"ContainerStarted","Data":"ba74884f3bc880eb9de3e94336479089e5e2b244d84d945510cae450cfc074f7"}
Sep 16 18:34:35.278900 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:35.278817    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" podStartSLOduration=2.2787913890000002 podStartE2EDuration="2.278791389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:34:35.273157812 +0000 UTC m=+1113.138061861" watchObservedRunningTime="2026-09-16 18:34:35.278791389 +0000 UTC m=+1113.143695437"
Sep 16 18:34:35.329080 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:35.328982    2562 patch_prober.go:28] interesting pod/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 18:34:35.329080 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:35.329063    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np" podUID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 18:34:35.803229 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:35.803195    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:34:35.859356 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:35.859319    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:35.876133 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:35.876094    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:36.049105 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:36.049059    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:36.061182 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:36.061089    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:34:36.329051 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:36.328962    2562 patch_prober.go:28] interesting pod/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 18:34:36.329051 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:36.329027    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np" podUID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 18:34:37.328775 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:37.328722    2562 patch_prober.go:28] interesting pod/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 18:34:37.329175 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:37.328797    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np" podUID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 18:34:38.052672 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:38.052634    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:38.328149 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:38.328039    2562 patch_prober.go:28] interesting pod/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 18:34:38.328149 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:38.328116    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np" podUID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 18:34:38.389606 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:38.389506    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:38.664656 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:38.664510    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:39.328570 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:39.328513    2562 patch_prober.go:28] interesting pod/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 18:34:39.328856 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:39.328826    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np" podUID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 18:34:40.135588 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:40.135186    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:40.328342 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:40.328303    2562 patch_prober.go:28] interesting pod/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 16 18:34:40.328575 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:40.328369    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np" podUID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 16 18:34:41.332920 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:41.332888    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:41.334054 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:41.334029    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:41.334928 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:41.334907    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:34:42.708264 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:42.708226    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:43.836401 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:43.836321    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:44.913713 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:44.913669    2562 patch_prober.go:28] interesting pod/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 18:34:44.914109 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:44.913761    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27" podUID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 18:34:45.693588 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:45.693526    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:45.920751 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:45.920649    2562 patch_prober.go:28] interesting pod/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 18:34:45.920751 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:45.920724    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh" podUID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 18:34:45.927498 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:45.927470    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:46.147658 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:46.147624    2562 generic.go:356] "Generic (PLEG): container finished" podID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerID="8b3410298278bdf3ecf5deb02d715dde4dd3a69993c80d3a4e2fdc7602f51575" exitCode=0
Sep 16 18:34:46.147814 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:46.147701    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27" event={"ID":"81ae555d-de1f-4a84-9788-84ffc4816fd9","Type":"ContainerDied","Data":"8b3410298278bdf3ecf5deb02d715dde4dd3a69993c80d3a4e2fdc7602f51575"}
Sep 16 18:34:46.148349 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:46.148330    2562 scope.go:117] "RemoveContainer" containerID="8b3410298278bdf3ecf5deb02d715dde4dd3a69993c80d3a4e2fdc7602f51575"
Sep 16 18:34:46.149896 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:46.149860    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:46.150002 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:46.149955    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:46.150042 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:46.150008    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:46.291811 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:46.291774    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:46.486865 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:46.486836    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:46.853309 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:46.853194    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:47.174628 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.173655    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerID="33f49ca09ec6ea51a537233df1753b70ec6baf0ef9ac1307545bed63bf7eff89" exitCode=0
Sep 16 18:34:47.174628 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.173837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh" event={"ID":"ec6d40bd-176d-4785-833d-d894891fbe1f","Type":"ContainerDied","Data":"33f49ca09ec6ea51a537233df1753b70ec6baf0ef9ac1307545bed63bf7eff89"}
Sep 16 18:34:47.174628 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.174376    2562 scope.go:117] "RemoveContainer" containerID="33f49ca09ec6ea51a537233df1753b70ec6baf0ef9ac1307545bed63bf7eff89"
Sep 16 18:34:47.179565 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.176691    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:47.179565 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.176798    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:47.179565 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.176848    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:47.180568 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.180312    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27" event={"ID":"81ae555d-de1f-4a84-9788-84ffc4816fd9","Type":"ContainerStarted","Data":"cd1429eca540283d131f6db43fc1f90bcaa48545fe66b65afc3b98ccd1038afc"}
Sep 16 18:34:47.196112 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.196084    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:47.270791 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.269126    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"]
Sep 16 18:34:47.284039 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.283948    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.287814 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.287783    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6ef84e-openshift-default-dockercfg-hnfgd\""
Sep 16 18:34:47.324322 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.324279    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2mwc\" (UniqueName: \"kubernetes.io/projected/befa9b40-d175-440e-b9c5-ff809620b697-kube-api-access-s2mwc\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.324519 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.324353    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befa9b40-d175-440e-b9c5-ff809620b697-istio-podinfo\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.324519 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.324386    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-workload-socket\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.324519 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.324411    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-istio-data\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.324519 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.324491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-istio-envoy\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.324770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.324560    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-credential-socket\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.324770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.324607    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befa9b40-d175-440e-b9c5-ff809620b697-istio-token\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.324770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.324639    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-workload-certs\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.324770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.324663    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befa9b40-d175-440e-b9c5-ff809620b697-istiod-ca-cert\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.426240 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.426138    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-credential-socket\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.426240 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.426184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befa9b40-d175-440e-b9c5-ff809620b697-istio-token\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.426240 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.426206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-workload-certs\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.426643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.426506    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-workload-certs\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.426643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.426593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befa9b40-d175-440e-b9c5-ff809620b697-istiod-ca-cert\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.426771 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.426697    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2mwc\" (UniqueName: \"kubernetes.io/projected/befa9b40-d175-440e-b9c5-ff809620b697-kube-api-access-s2mwc\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.426771 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.426749    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befa9b40-d175-440e-b9c5-ff809620b697-istio-podinfo\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.426771 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.426771    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-workload-socket\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.426933 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.426790    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-istio-data\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.426933 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.426819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-istio-envoy\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.427337 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.427283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befa9b40-d175-440e-b9c5-ff809620b697-istiod-ca-cert\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.427448 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.427351    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-workload-socket\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.427521 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.427461    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-istio-data\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.427802 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.427776    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-credential-socket\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.429482 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.429459    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-istio-envoy\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.430636 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.430614    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befa9b40-d175-440e-b9c5-ff809620b697-istio-podinfo\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.500582 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.498626    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"]
Sep 16 18:34:47.516702 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.516665    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befa9b40-d175-440e-b9c5-ff809620b697-istio-token\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.516890 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.516873    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2mwc\" (UniqueName: \"kubernetes.io/projected/befa9b40-d175-440e-b9c5-ff809620b697-kube-api-access-s2mwc\") pod \"e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") " pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.596225 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.596190    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:47.714686 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:47.714649    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:48.024172 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:34:48.024135    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbefa9b40_d175_440e_b9c5_ff809620b697.slice/crio-684d302511836dc4bf28093a7d56218aafce9785856b6eae4b384fc3b6ddf6ae WatchSource:0}: Error finding container 684d302511836dc4bf28093a7d56218aafce9785856b6eae4b384fc3b6ddf6ae: Status 404 returned error can't find the container with id 684d302511836dc4bf28093a7d56218aafce9785856b6eae4b384fc3b6ddf6ae
Sep 16 18:34:48.026710 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.026675    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:48.026797 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.026746    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:48.026797 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.026762    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"]
Sep 16 18:34:48.026797 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.026776    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:34:48.037394 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.037030    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:48.171576 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.171522    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:48.171795 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.171677    2562 patch_prober.go:28] interesting pod/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 18:34:48.171795 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.171726    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27" podUID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 18:34:48.197756 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.197713    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh" event={"ID":"ec6d40bd-176d-4785-833d-d894891fbe1f","Type":"ContainerStarted","Data":"984caac5dd48ed418cfe92b012102fec01607fc7a0777c5e0b6394baaeba5dd9"}
Sep 16 18:34:48.205704 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.205661    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz" event={"ID":"befa9b40-d175-440e-b9c5-ff809620b697","Type":"ContainerStarted","Data":"a373737b3f649b401fcf52529fe465e0f731c22f680631ba0bac72a054c41291"}
Sep 16 18:34:48.205909 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.205725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz" event={"ID":"befa9b40-d175-440e-b9c5-ff809620b697","Type":"ContainerStarted","Data":"684d302511836dc4bf28093a7d56218aafce9785856b6eae4b384fc3b6ddf6ae"}
Sep 16 18:34:48.441237 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.441083    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz" podStartSLOduration=1.4410595480000001 podStartE2EDuration="1.441059548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:34:48.439853664 +0000 UTC m=+1126.304757724" watchObservedRunningTime="2026-09-16 18:34:48.441059548 +0000 UTC m=+1126.305963596"
Sep 16 18:34:48.472569 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.469724    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:48.472569 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.470136    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:48.472569 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.470198    2562 patch_prober.go:28] interesting pod/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 18:34:48.472569 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.470270    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh" podUID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 18:34:48.597354 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.597306    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:48.799769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.799625    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:48.995179 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:48.995100    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:49.171067 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:49.170957    2562 patch_prober.go:28] interesting pod/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 18:34:49.171067 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:49.171032    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27" podUID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 18:34:49.468570 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:49.467428    2562 patch_prober.go:28] interesting pod/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 18:34:49.468570 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:49.467502    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh" podUID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 18:34:49.604396 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:49.604360    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:50.171390 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:50.171338    2562 patch_prober.go:28] interesting pod/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 18:34:50.171688 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:50.171415    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27" podUID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 18:34:50.216839 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:50.216807    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:50.218065 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:50.218042    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:34:50.469569 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:50.466985    2562 patch_prober.go:28] interesting pod/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 18:34:50.469569 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:50.467058    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh" podUID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 18:34:50.633730 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:50.633673    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:51.171313 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:51.171263    2562 patch_prober.go:28] interesting pod/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 18:34:51.171529 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:51.171341    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27" podUID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 18:34:51.193576 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:51.193520    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:51.466878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:51.466831    2562 patch_prober.go:28] interesting pod/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 18:34:51.467102 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:51.466920    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh" podUID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 18:34:51.540774 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:51.540738    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:51.879404 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:51.879298    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:52.170830 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:52.170723    2562 patch_prober.go:28] interesting pod/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 16 18:34:52.170830 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:52.170806    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27" podUID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 16 18:34:52.467568 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:52.467506    2562 patch_prober.go:28] interesting pod/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 18:34:52.467765 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:52.467592    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh" podUID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 18:34:53.177167 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:53.177121    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:53.177596 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:53.177577    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:53.178406 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:53.178386    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:34:53.469008 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:53.468963    2562 patch_prober.go:28] interesting pod/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 16 18:34:53.469300 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:53.469262    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh" podUID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 16 18:34:54.470887 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:54.470848    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:54.471343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:54.471232    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:54.471924 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:54.471906    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:34:55.657030 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:55.656989    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:55.841098 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:55.841062    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:56.022909 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:56.022871    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:57.455042 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:57.454997    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:57.638830 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:57.638788    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:58.728907 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:34:58.728869    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:00.856936 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:00.856898    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:35:01.051924 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:01.051876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:03.266567 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:03.266401    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:03.529967 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:03.529876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:03.920927 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:03.920836    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:04.087447 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:04.087402    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:05.884496 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:05.884448    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:06.027349 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:06.027317    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:07.127780 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:07.127743    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:07.265568 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:07.265238    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:08.313176 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:08.313134    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:10.987900 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:10.987858    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:11.360481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:11.360449    2562 generic.go:356] "Generic (PLEG): container finished" podID="2e520ad7-28d2-4a42-8bf3-30e3159a53f9" containerID="85858c37c4e91192fb10fe9ecc8d21b23675b2840d8ae50a2a9b1db2818c61c5" exitCode=0
Sep 16 18:35:11.360725 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:11.360525    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp" event={"ID":"2e520ad7-28d2-4a42-8bf3-30e3159a53f9","Type":"ContainerDied","Data":"85858c37c4e91192fb10fe9ecc8d21b23675b2840d8ae50a2a9b1db2818c61c5"}
Sep 16 18:35:11.361114 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:11.361083    2562 scope.go:117] "RemoveContainer" containerID="85858c37c4e91192fb10fe9ecc8d21b23675b2840d8ae50a2a9b1db2818c61c5"
Sep 16 18:35:11.362484 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:11.362465    2562 generic.go:356] "Generic (PLEG): container finished" podID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerID="235b0d4e0a10ef343a35f9c032acd9d4bf0b33c08885cf4b6cae408bf6d70e2b" exitCode=0
Sep 16 18:35:11.362586 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:11.362511    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" event={"ID":"7ee0c731-debf-4fc9-afc2-8f8a54d279ca","Type":"ContainerDied","Data":"235b0d4e0a10ef343a35f9c032acd9d4bf0b33c08885cf4b6cae408bf6d70e2b"}
Sep 16 18:35:11.362586 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:11.362524    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:35:11.362670 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:11.362620    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:35:11.362704 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:11.362666    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:35:11.363058 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:11.363043    2562 scope.go:117] "RemoveContainer" containerID="235b0d4e0a10ef343a35f9c032acd9d4bf0b33c08885cf4b6cae408bf6d70e2b"
Sep 16 18:35:11.364331 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:11.364300    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:35:11.364396 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:11.364383    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:35:11.364456 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:11.364427    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:35:12.376674 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:12.376627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp" event={"ID":"2e520ad7-28d2-4a42-8bf3-30e3159a53f9","Type":"ContainerStarted","Data":"4359c963fa6c405a8a5c0c46b6f0a62086f580f4e61b972866d8b535a902e86c"}
Sep 16 18:35:12.380505 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:12.380472    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" event={"ID":"7ee0c731-debf-4fc9-afc2-8f8a54d279ca","Type":"ContainerStarted","Data":"f5c9600bd726d9bae5bfadd9510b0039178589b62d08bf28e1f5635f55b381cd"}
Sep 16 18:35:12.818713 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:12.818673    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:12.859000 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:12.858962    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:35:12.860062 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:12.860025    2562 patch_prober.go:28] interesting pod/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 16 18:35:12.860359 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:12.860313    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 16 18:35:12.990741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:12.990701    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:35:13.312167 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:13.312124    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:35:13.313380 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:13.313354    2562 patch_prober.go:28] interesting pod/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 16 18:35:13.313509 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:13.313405    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp" podUID="2e520ad7-28d2-4a42-8bf3-30e3159a53f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 16 18:35:13.858119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:13.858070    2562 patch_prober.go:28] interesting pod/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 16 18:35:13.858636 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:13.858144    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 16 18:35:14.312302 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:14.312259    2562 patch_prober.go:28] interesting pod/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 16 18:35:14.312521 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:14.312322    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp" podUID="2e520ad7-28d2-4a42-8bf3-30e3159a53f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 16 18:35:14.762109 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:14.762073    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:14.857601 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:14.857560    2562 patch_prober.go:28] interesting pod/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 16 18:35:14.857917 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:14.857888    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 16 18:35:15.312400 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:15.312357    2562 patch_prober.go:28] interesting pod/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 16 18:35:15.312894 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:15.312423    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp" podUID="2e520ad7-28d2-4a42-8bf3-30e3159a53f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 16 18:35:15.654943 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:15.654809    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:15.857399 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:15.857354    2562 patch_prober.go:28] interesting pod/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 16 18:35:15.857739 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:15.857697    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 16 18:35:16.312378 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:16.312333    2562 patch_prober.go:28] interesting pod/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 16 18:35:16.312618 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:16.312408    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp" podUID="2e520ad7-28d2-4a42-8bf3-30e3159a53f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 16 18:35:16.857134 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:16.857086    2562 patch_prober.go:28] interesting pod/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 16 18:35:16.857363 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:16.857160    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 16 18:35:17.071930 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:17.071884    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:17.312071 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:17.312024    2562 patch_prober.go:28] interesting pod/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 16 18:35:17.312288 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:17.312111    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp" podUID="2e520ad7-28d2-4a42-8bf3-30e3159a53f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 16 18:35:17.799759 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:17.799711    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:17.857421 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:17.857368    2562 patch_prober.go:28] interesting pod/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 16 18:35:17.857666 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:17.857445    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 16 18:35:17.999146 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:17.999092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:18.316620 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:18.316592    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:35:18.317894 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:18.317868    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:35:18.857985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:18.857943    2562 patch_prober.go:28] interesting pod/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 16 18:35:18.858492 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:18.858035    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 16 18:35:19.379853 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:19.379807    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:19.800746 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:19.800714    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h"]
Sep 16 18:35:19.803796 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:19.803777    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h"
Sep 16 18:35:19.812366 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:35:19.812326    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-ait-119aafd2-mff2h\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" podUID="5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h"
Sep 16 18:35:19.812366 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:35:19.812346    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-cleanup-dockercfg-cgwrl\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\"" type="*v1.Secret"
Sep 16 18:35:19.812514 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:35:19.812450    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 16 18:35:19.862797 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:19.862755    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h"]
Sep 16 18:35:19.886716 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:19.886683    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:35:19.887772 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:19.887702    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:35:19.888525 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:19.888501    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:35:19.927785 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:19.927749    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-119aafd2-mff2h\" (UID: \"5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h"
Sep 16 18:35:20.028664 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:20.028623    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-119aafd2-mff2h\" (UID: \"5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h"
Sep 16 18:35:21.019762 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:21.019726    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:21.029593 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:35:21.029568    2562 configmap.go:193] Couldn't get configMap odh-ai-gateway-infra/openshift-service-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 16 18:35:21.029727 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:35:21.029644    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999-maas-api-service-ca podName:5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999 nodeName:}" failed. No retries permitted until 2026-09-16 18:35:21.529628331 +0000 UTC m=+1159.394532360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-service-ca" (UniqueName: "kubernetes.io/configmap/5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999-maas-api-service-ca") pod "maas-api-revoke-keys-e2e-ait-119aafd2-mff2h" (UID: "5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999") : failed to sync configmap cache: timed out waiting for the condition
Sep 16 18:35:21.084754 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:21.084723    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:35:21.292578 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:21.292462    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:35:21.441919 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:21.441886    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:21.544725 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:21.544627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-119aafd2-mff2h\" (UID: \"5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h"
Sep 16 18:35:21.545261 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:21.545240    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-119aafd2-mff2h\" (UID: \"5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h"
Sep 16 18:35:21.613813 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:21.613780    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h"
Sep 16 18:35:21.924671 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:21.924642    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h"]
Sep 16 18:35:21.926258 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:35:21.926227    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac70a0d_74e6_40fd_9bc6_c5a7b24fd999.slice/crio-f66dd4f23e0166ea14b5e602b9e1f9079fe4f8d543d97ab1990143ffdc74d51c WatchSource:0}: Error finding container f66dd4f23e0166ea14b5e602b9e1f9079fe4f8d543d97ab1990143ffdc74d51c: Status 404 returned error can't find the container with id f66dd4f23e0166ea14b5e602b9e1f9079fe4f8d543d97ab1990143ffdc74d51c
Sep 16 18:35:22.479536 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:22.479490    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h" event={"ID":"5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999","Type":"ContainerStarted","Data":"f66dd4f23e0166ea14b5e602b9e1f9079fe4f8d543d97ab1990143ffdc74d51c"}
Sep 16 18:35:23.173200 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:23.173165    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:24.496199 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:24.496152    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h" event={"ID":"5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999","Type":"ContainerStarted","Data":"b9326e066fe656a666b74c80bb3c2557348fcd3c1e0d61fc12e07fcef1fabd4f"}
Sep 16 18:35:24.541729 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:24.541677    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h" podStartSLOduration=3.403576105 podStartE2EDuration="5.541659462s" totalImagesPullingTime="2.138083357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:35:24.537108713 +0000 UTC m=+1162.402012761" watchObservedRunningTime="2026-09-16 18:35:24.541659462 +0000 UTC m=+1162.406563512"
Sep 16 18:35:24.880337 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:24.880257    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:25.504139 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:25.504107    2562 generic.go:356] "Generic (PLEG): container finished" podID="5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999" containerID="b9326e066fe656a666b74c80bb3c2557348fcd3c1e0d61fc12e07fcef1fabd4f" exitCode=0
Sep 16 18:35:25.504587 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:25.504181    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h" event={"ID":"5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999","Type":"ContainerDied","Data":"b9326e066fe656a666b74c80bb3c2557348fcd3c1e0d61fc12e07fcef1fabd4f"}
Sep 16 18:35:26.518689 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:26.518643    2562 generic.go:356] "Generic (PLEG): container finished" podID="befa9b40-d175-440e-b9c5-ff809620b697" containerID="a373737b3f649b401fcf52529fe465e0f731c22f680631ba0bac72a054c41291" exitCode=0
Sep 16 18:35:26.519222 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:26.518690    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz" event={"ID":"befa9b40-d175-440e-b9c5-ff809620b697","Type":"ContainerDied","Data":"a373737b3f649b401fcf52529fe465e0f731c22f680631ba0bac72a054c41291"}
Sep 16 18:35:26.519646 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:26.519609    2562 scope.go:117] "RemoveContainer" containerID="a373737b3f649b401fcf52529fe465e0f731c22f680631ba0bac72a054c41291"
Sep 16 18:35:26.520956 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:26.520922    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:35:26.521036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:26.521008    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:35:26.521076 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:26.521052    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:35:26.685612 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:26.685584    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h"
Sep 16 18:35:26.800630 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:26.800536    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999-maas-api-service-ca\") pod \"5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999\" (UID: \"5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999\") "
Sep 16 18:35:26.800884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:26.800860    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999" (UID: "5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:35:26.800936 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:26.800928    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999-maas-api-service-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:35:26.841267 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:26.841237    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:27.526918 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:27.526886    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz" event={"ID":"befa9b40-d175-440e-b9c5-ff809620b697","Type":"ContainerStarted","Data":"5a4e2d9fae6f038fb6ddcccb29ee1ad725ae432cea9aba92ec07b8762a0f8c51"}
Sep 16 18:35:27.528150 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:27.528129    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h"
Sep 16 18:35:27.528263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:27.528167    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h" event={"ID":"5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999","Type":"ContainerDied","Data":"f66dd4f23e0166ea14b5e602b9e1f9079fe4f8d543d97ab1990143ffdc74d51c"}
Sep 16 18:35:27.528263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:27.528203    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f66dd4f23e0166ea14b5e602b9e1f9079fe4f8d543d97ab1990143ffdc74d51c"
Sep 16 18:35:27.597103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:27.597068    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:35:27.597394 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:27.597372    2562 patch_prober.go:28] interesting pod/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 18:35:27.597439 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:27.597421    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz" podUID="befa9b40-d175-440e-b9c5-ff809620b697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 18:35:28.080226 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:28.080180    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:35:28.597002 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:28.596962    2562 patch_prober.go:28] interesting pod/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 18:35:28.597421 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:28.597017    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz" podUID="befa9b40-d175-440e-b9c5-ff809620b697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 18:35:29.083255 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:29.083204    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:29.597108 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:29.597075    2562 patch_prober.go:28] interesting pod/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 18:35:29.597501 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:29.597139    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz" podUID="befa9b40-d175-440e-b9c5-ff809620b697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 18:35:30.586343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:30.586303    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:30.597424 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:30.597399    2562 patch_prober.go:28] interesting pod/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 18:35:30.597778 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:30.597447    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz" podUID="befa9b40-d175-440e-b9c5-ff809620b697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 18:35:31.596883 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:31.596843    2562 patch_prober.go:28] interesting pod/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 18:35:31.597068 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:31.596901    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz" podUID="befa9b40-d175-440e-b9c5-ff809620b697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 18:35:31.833884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:31.833847    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:32.596988 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:32.596950    2562 patch_prober.go:28] interesting pod/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Sep 16 18:35:32.597167 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:32.597012    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz" podUID="befa9b40-d175-440e-b9c5-ff809620b697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Sep 16 18:35:33.134446 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:33.134414    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:33.277652 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:33.277609    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:33.398556 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:33.398458    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:33.590336 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:33.590123    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:33.603676 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:33.603643    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:35:33.604088 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:33.604064    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:35:33.604829 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:33.604805    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:35:33.706056 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:33.706020    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h"]
Sep 16 18:35:33.715057 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:33.715029    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-119aafd2-mff2h"]
Sep 16 18:35:33.812592 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:33.812535    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:33.980097 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:33.980013    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:34.032448 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:34.032417    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"]
Sep 16 18:35:34.032648 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:34.032625    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerName="istio-proxy" containerID="cri-o://f5c9600bd726d9bae5bfadd9510b0039178589b62d08bf28e1f5635f55b381cd" gracePeriod=30
Sep 16 18:35:34.538382 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:34.538351    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:34.551422 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:34.551389    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999" path="/var/lib/kubelet/pods/5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999/volumes"
Sep 16 18:35:34.752707 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:34.752671    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:34.889712 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:34.889609    2562 patch_prober.go:28] interesting pod/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 18:35:34.889712 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:34.889694    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:35:35.004203 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.004169    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:35.650786 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.650749    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:35.656986 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.656957    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"]
Sep 16 18:35:35.657399 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.657386    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999" containerName="revoke-keys"
Sep 16 18:35:35.657443 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.657401    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999" containerName="revoke-keys"
Sep 16 18:35:35.657490 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.657481    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac70a0d-74e6-40fd-9bc6-c5a7b24fd999" containerName="revoke-keys"
Sep 16 18:35:35.663964 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.663945    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.674626 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.674606    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ad2bfa56-gw-openshift-default-dockercfg-rf8fl\""
Sep 16 18:35:35.703918 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.703883    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-podinfo\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.704046 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.704001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istiod-ca-cert\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.704046 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.704038    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-credential-socket\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.704157 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.704065    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-workload-socket\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.704219 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.704176    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-workload-certs\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.704219 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.704211    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-envoy\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.704324 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.704248    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpxp6\" (UniqueName: \"kubernetes.io/projected/6c8708e5-0029-4ffe-b4e1-65301c07efb8-kube-api-access-qpxp6\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.704324 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.704306    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-token\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.704423 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.704338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-data\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.757444 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.757414    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"]
Sep 16 18:35:35.805420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.805387    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-podinfo\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.805648 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.805444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istiod-ca-cert\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.805648 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.805469    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-credential-socket\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.805648 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.805493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-workload-socket\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.805648 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.805523    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-workload-certs\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.805648 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.805538    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-envoy\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.805648 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.805596    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpxp6\" (UniqueName: \"kubernetes.io/projected/6c8708e5-0029-4ffe-b4e1-65301c07efb8-kube-api-access-qpxp6\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.805648 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.805639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-token\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.805999 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.805669    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-data\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.805999 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.805944    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-workload-certs\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.806112 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.806021    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-credential-socket\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.806158 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.806098    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-data\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.806213 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.806194    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-workload-socket\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.806262 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.806236    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istiod-ca-cert\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.808059 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.808037    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-envoy\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.808177 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.808162    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-podinfo\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.843014 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.842988    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpxp6\" (UniqueName: \"kubernetes.io/projected/6c8708e5-0029-4ffe-b4e1-65301c07efb8-kube-api-access-qpxp6\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.844698 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.844679    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-token\") pod \"e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") " pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:35.974698 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:35.974655    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:36.306617 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:36.306590    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"]
Sep 16 18:35:36.307911 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:35:36.307883    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8708e5_0029_4ffe_b4e1_65301c07efb8.slice/crio-d3aec1b5e8dd6920a31430b924236f6b22e663e79400df17d8350d83a38be871 WatchSource:0}: Error finding container d3aec1b5e8dd6920a31430b924236f6b22e663e79400df17d8350d83a38be871: Status 404 returned error can't find the container with id d3aec1b5e8dd6920a31430b924236f6b22e663e79400df17d8350d83a38be871
Sep 16 18:35:36.310088 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:36.310058    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:36.310340 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:36.310310    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:35:36.310388 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:36.310374    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:35:36.310429 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:36.310407    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:35:36.598179 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:36.598097    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht" event={"ID":"6c8708e5-0029-4ffe-b4e1-65301c07efb8","Type":"ContainerStarted","Data":"f0db9a262d32efb570bc8c5907270d65a81f09441d671beb20ae8cf24ce9e43e"}
Sep 16 18:35:36.598179 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:36.598158    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht" event={"ID":"6c8708e5-0029-4ffe-b4e1-65301c07efb8","Type":"ContainerStarted","Data":"d3aec1b5e8dd6920a31430b924236f6b22e663e79400df17d8350d83a38be871"}
Sep 16 18:35:36.757672 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:36.757608    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht" podStartSLOduration=1.757591034 podStartE2EDuration="1.757591034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:35:36.747616056 +0000 UTC m=+1174.612520105" watchObservedRunningTime="2026-09-16 18:35:36.757591034 +0000 UTC m=+1174.622495176"
Sep 16 18:35:36.975235 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:36.975199    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:37.981431 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:37.981397    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:38.398761 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.398678    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:38.609708 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.609676    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:38.610571 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.610523    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:35:38.739429 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.739390    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:38.756421 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.756392    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"]
Sep 16 18:35:38.759029 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.759011    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.773188 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.766402    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-909a40\"/\"kube-root-ca.crt\""
Sep 16 18:35:38.773188 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.766665    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-909a40\"/\"openshift-service-ca.crt\""
Sep 16 18:35:38.773188 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.767158    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-909a40\"/\"simulator-w0-909a40-kserve-self-signed-certs\""
Sep 16 18:35:38.781262 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.781243    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-909a40\"/\"default-dockercfg-rqmcw\""
Sep 16 18:35:38.823197 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.823155    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"]
Sep 16 18:35:38.835275 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.835241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-model-cache\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.835454 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.835290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-home\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.835454 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.835348    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-tmp-dir\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.835454 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.835398    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-dshm\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.835454 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.835438    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb505d0-3913-4ebe-b987-b80c019a2f47-tls-certs\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.835730 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.835516    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m569\" (UniqueName: \"kubernetes.io/projected/bdb505d0-3913-4ebe-b987-b80c019a2f47-kube-api-access-9m569\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.907436 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.907335    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:38.936089 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.936052    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m569\" (UniqueName: \"kubernetes.io/projected/bdb505d0-3913-4ebe-b987-b80c019a2f47-kube-api-access-9m569\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.936267 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.936136    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-model-cache\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.936267 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.936183    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-home\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.936267 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.936205    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-tmp-dir\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.936267 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.936228    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-dshm\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.936267 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.936265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb505d0-3913-4ebe-b987-b80c019a2f47-tls-certs\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.936638 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.936610    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-model-cache\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.936735 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.936636    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-tmp-dir\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.936735 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.936690    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-home\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.938643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.938621    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-dshm\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.938878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.938861    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb505d0-3913-4ebe-b987-b80c019a2f47-tls-certs\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:38.964935 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:38.964900    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m569\" (UniqueName: \"kubernetes.io/projected/bdb505d0-3913-4ebe-b987-b80c019a2f47-kube-api-access-9m569\") pod \"simulator-w0-909a40-kserve-6669d85d76-cqtqf\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") " pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:39.076167 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.076135    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:39.297618 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.297535    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:35:39.340062 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340035    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-data\") pod \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") "
Sep 16 18:35:39.340062 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340068    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-workload-certs\") pod \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") "
Sep 16 18:35:39.340299 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340088    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-token\") pod \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") "
Sep 16 18:35:39.340299 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340130    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-podinfo\") pod \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") "
Sep 16 18:35:39.340299 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340191    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-envoy\") pod \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") "
Sep 16 18:35:39.340299 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340221    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-workload-socket\") pod \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") "
Sep 16 18:35:39.340299 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340291    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-credential-socket\") pod \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") "
Sep 16 18:35:39.340591 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340311    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "7ee0c731-debf-4fc9-afc2-8f8a54d279ca" (UID: "7ee0c731-debf-4fc9-afc2-8f8a54d279ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:35:39.340591 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340323    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhq4f\" (UniqueName: \"kubernetes.io/projected/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-kube-api-access-dhq4f\") pod \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") "
Sep 16 18:35:39.340591 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340384    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istiod-ca-cert\") pod \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\" (UID: \"7ee0c731-debf-4fc9-afc2-8f8a54d279ca\") "
Sep 16 18:35:39.340591 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340334    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7ee0c731-debf-4fc9-afc2-8f8a54d279ca" (UID: "7ee0c731-debf-4fc9-afc2-8f8a54d279ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:35:39.340809 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340597    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7ee0c731-debf-4fc9-afc2-8f8a54d279ca" (UID: "7ee0c731-debf-4fc9-afc2-8f8a54d279ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:35:39.340809 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340655    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7ee0c731-debf-4fc9-afc2-8f8a54d279ca" (UID: "7ee0c731-debf-4fc9-afc2-8f8a54d279ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:35:39.340949 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340926    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-workload-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:35:39.341008 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340954    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-credential-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:35:39.341008 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340968    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-data\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:35:39.341008 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340991    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-workload-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:35:39.341008 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.340966    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7ee0c731-debf-4fc9-afc2-8f8a54d279ca" (UID: "7ee0c731-debf-4fc9-afc2-8f8a54d279ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:35:39.342427 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.342404    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7ee0c731-debf-4fc9-afc2-8f8a54d279ca" (UID: "7ee0c731-debf-4fc9-afc2-8f8a54d279ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:35:39.342526 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.342401    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7ee0c731-debf-4fc9-afc2-8f8a54d279ca" (UID: "7ee0c731-debf-4fc9-afc2-8f8a54d279ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:35:39.342719 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.342703    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "7ee0c731-debf-4fc9-afc2-8f8a54d279ca" (UID: "7ee0c731-debf-4fc9-afc2-8f8a54d279ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:35:39.342769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.342716    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-kube-api-access-dhq4f" (OuterVolumeSpecName: "kube-api-access-dhq4f") pod "7ee0c731-debf-4fc9-afc2-8f8a54d279ca" (UID: "7ee0c731-debf-4fc9-afc2-8f8a54d279ca"). InnerVolumeSpecName "kube-api-access-dhq4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:35:39.437905 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:35:39.437874    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdb505d0_3913_4ebe_b987_b80c019a2f47.slice/crio-c510a924ed14bc01bbca725be50dac93ee1847b9266472c1190b2d4947acfaef WatchSource:0}: Error finding container c510a924ed14bc01bbca725be50dac93ee1847b9266472c1190b2d4947acfaef: Status 404 returned error can't find the container with id c510a924ed14bc01bbca725be50dac93ee1847b9266472c1190b2d4947acfaef
Sep 16 18:35:39.441979 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.441958    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-envoy\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:35:39.442065 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.441982    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhq4f\" (UniqueName: \"kubernetes.io/projected/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-kube-api-access-dhq4f\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:35:39.442065 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.441993    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istiod-ca-cert\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:35:39.442065 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.442002    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-token\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:35:39.442065 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.442010    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee0c731-debf-4fc9-afc2-8f8a54d279ca-istio-podinfo\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:35:39.442783 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.442762    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"]
Sep 16 18:35:39.617246 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.617160    2562 generic.go:356] "Generic (PLEG): container finished" podID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerID="f5c9600bd726d9bae5bfadd9510b0039178589b62d08bf28e1f5635f55b381cd" exitCode=0
Sep 16 18:35:39.617246 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.617211    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"
Sep 16 18:35:39.617246 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.617240    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" event={"ID":"7ee0c731-debf-4fc9-afc2-8f8a54d279ca","Type":"ContainerDied","Data":"f5c9600bd726d9bae5bfadd9510b0039178589b62d08bf28e1f5635f55b381cd"}
Sep 16 18:35:39.617491 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.617292    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79" event={"ID":"7ee0c731-debf-4fc9-afc2-8f8a54d279ca","Type":"ContainerDied","Data":"ba74884f3bc880eb9de3e94336479089e5e2b244d84d945510cae450cfc074f7"}
Sep 16 18:35:39.617491 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.617324    2562 scope.go:117] "RemoveContainer" containerID="f5c9600bd726d9bae5bfadd9510b0039178589b62d08bf28e1f5635f55b381cd"
Sep 16 18:35:39.619184 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.619159    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf" event={"ID":"bdb505d0-3913-4ebe-b987-b80c019a2f47","Type":"ContainerStarted","Data":"893e02410559310374758e3046fa5cdb25a21a0b561b9a858a3482c6a2661e14"}
Sep 16 18:35:39.619311 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.619195    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf" event={"ID":"bdb505d0-3913-4ebe-b987-b80c019a2f47","Type":"ContainerStarted","Data":"c510a924ed14bc01bbca725be50dac93ee1847b9266472c1190b2d4947acfaef"}
Sep 16 18:35:39.628232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.628214    2562 scope.go:117] "RemoveContainer" containerID="235b0d4e0a10ef343a35f9c032acd9d4bf0b33c08885cf4b6cae408bf6d70e2b"
Sep 16 18:35:39.637006 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.636987    2562 scope.go:117] "RemoveContainer" containerID="f5c9600bd726d9bae5bfadd9510b0039178589b62d08bf28e1f5635f55b381cd"
Sep 16 18:35:39.637262 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:35:39.637243    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c9600bd726d9bae5bfadd9510b0039178589b62d08bf28e1f5635f55b381cd\": container with ID starting with f5c9600bd726d9bae5bfadd9510b0039178589b62d08bf28e1f5635f55b381cd not found: ID does not exist" containerID="f5c9600bd726d9bae5bfadd9510b0039178589b62d08bf28e1f5635f55b381cd"
Sep 16 18:35:39.637328 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.637268    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c9600bd726d9bae5bfadd9510b0039178589b62d08bf28e1f5635f55b381cd"} err="failed to get container status \"f5c9600bd726d9bae5bfadd9510b0039178589b62d08bf28e1f5635f55b381cd\": rpc error: code = NotFound desc = could not find container \"f5c9600bd726d9bae5bfadd9510b0039178589b62d08bf28e1f5635f55b381cd\": container with ID starting with f5c9600bd726d9bae5bfadd9510b0039178589b62d08bf28e1f5635f55b381cd not found: ID does not exist"
Sep 16 18:35:39.637328 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.637285    2562 scope.go:117] "RemoveContainer" containerID="235b0d4e0a10ef343a35f9c032acd9d4bf0b33c08885cf4b6cae408bf6d70e2b"
Sep 16 18:35:39.637581 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:35:39.637532    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235b0d4e0a10ef343a35f9c032acd9d4bf0b33c08885cf4b6cae408bf6d70e2b\": container with ID starting with 235b0d4e0a10ef343a35f9c032acd9d4bf0b33c08885cf4b6cae408bf6d70e2b not found: ID does not exist" containerID="235b0d4e0a10ef343a35f9c032acd9d4bf0b33c08885cf4b6cae408bf6d70e2b"
Sep 16 18:35:39.637654 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.637591    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235b0d4e0a10ef343a35f9c032acd9d4bf0b33c08885cf4b6cae408bf6d70e2b"} err="failed to get container status \"235b0d4e0a10ef343a35f9c032acd9d4bf0b33c08885cf4b6cae408bf6d70e2b\": rpc error: code = NotFound desc = could not find container \"235b0d4e0a10ef343a35f9c032acd9d4bf0b33c08885cf4b6cae408bf6d70e2b\": container with ID starting with 235b0d4e0a10ef343a35f9c032acd9d4bf0b33c08885cf4b6cae408bf6d70e2b not found: ID does not exist"
Sep 16 18:35:39.702204 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.702142    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf" podStartSLOduration=1.702126093 podStartE2EDuration="1.702126093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:35:39.699460636 +0000 UTC m=+1177.564364683" watchObservedRunningTime="2026-09-16 18:35:39.702126093 +0000 UTC m=+1177.567030140"
Sep 16 18:35:39.740371 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.740330    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"]
Sep 16 18:35:39.783143 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:39.783103    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-119aafd2-openshift-default-8599c89cc4-47k79"]
Sep 16 18:35:40.552474 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:40.552441    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" path="/var/lib/kubelet/pods/7ee0c731-debf-4fc9-afc2-8f8a54d279ca/volumes"
Sep 16 18:35:40.794575 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:40.794493    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:42.296673 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:42.296641    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:43.671504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:43.671461    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:45.103473 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:45.103438    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:45.952802 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:45.952762    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:35:47.741818 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:47.741781    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:48.061760 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:48.061673    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:48.719150 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:48.719119    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:48.847601 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:48.847567    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:48.986404 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:48.984213    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:49.076438 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.076403    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:49.076611 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.076464    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:49.089280 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.089252    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:49.389538 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.389447    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"]
Sep 16 18:35:49.389896 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.389882    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerName="istio-proxy"
Sep 16 18:35:49.389941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.389898    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerName="istio-proxy"
Sep 16 18:35:49.389941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.389913    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerName="istio-proxy"
Sep 16 18:35:49.389941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.389918    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerName="istio-proxy"
Sep 16 18:35:49.390038 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.389985    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerName="istio-proxy"
Sep 16 18:35:49.393061 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.393033    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.397285 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.397262    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d8a600\"/\"openshift-service-ca.crt\""
Sep 16 18:35:49.397426 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.397267    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d8a600\"/\"kube-root-ca.crt\""
Sep 16 18:35:49.397426 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.397300    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d8a600\"/\"default-dockercfg-vk6zb\""
Sep 16 18:35:49.398299 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.398276    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d8a600\"/\"simulator-w4-d8a600-kserve-self-signed-certs\""
Sep 16 18:35:49.451088 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.451055    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"]
Sep 16 18:35:49.546835 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.546797    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-home\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.546835 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.546837    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ft77\" (UniqueName: \"kubernetes.io/projected/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-kube-api-access-8ft77\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.547049 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.546866    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-model-cache\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.547049 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.547002    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-tmp-dir\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.547049 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.547032    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-dshm\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.547153 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.547058    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-tls-certs\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.648466 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.648377    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-tmp-dir\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.648466 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.648415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-dshm\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.648466 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.648435    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-tls-certs\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.648796 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.648495    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-home\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.648796 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.648510    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ft77\" (UniqueName: \"kubernetes.io/projected/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-kube-api-access-8ft77\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.648796 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.648530    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-model-cache\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.648971 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.648872    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-tmp-dir\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.649032 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.648989    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-model-cache\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.649141 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.649111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-home\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.650983 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.650951    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-dshm\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.651190 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.651170    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-tls-certs\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.683242 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.683215    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ft77\" (UniqueName: \"kubernetes.io/projected/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-kube-api-access-8ft77\") pod \"simulator-w4-d8a600-kserve-67cfc9f79b-lgngd\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") " pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.702337 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.702308    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:49.703774 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.703745    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:35:49.728465 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:49.728431    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:50.027699 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:50.027670    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"]
Sep 16 18:35:50.029307 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:35:50.029279    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee7d27e9_1bbf_429f_b3b2_052f8e704fb3.slice/crio-dbc88887cc8e75db3852df767eeffce9d1da631b1881a1416e838e7ace458065 WatchSource:0}: Error finding container dbc88887cc8e75db3852df767eeffce9d1da631b1881a1416e838e7ace458065: Status 404 returned error can't find the container with id dbc88887cc8e75db3852df767eeffce9d1da631b1881a1416e838e7ace458065
Sep 16 18:35:50.115800 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:50.115743    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:50.700481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:50.700443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd" event={"ID":"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3","Type":"ContainerStarted","Data":"728505d1be4c8a519cdae425e2f47cba5db1a5ed17cb4732f43f7d68993e7618"}
Sep 16 18:35:50.700646 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:50.700496    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd" event={"ID":"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3","Type":"ContainerStarted","Data":"dbc88887cc8e75db3852df767eeffce9d1da631b1881a1416e838e7ace458065"}
Sep 16 18:35:51.227250 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:51.227216    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:59.702686 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:59.702637    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:59.702686 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:59.702696    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:59.715240 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:59.715208    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:35:59.744051 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:59.743996    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd" podStartSLOduration=10.743981797 podStartE2EDuration="10.743981797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:35:50.751376747 +0000 UTC m=+1188.616280795" watchObservedRunningTime="2026-09-16 18:35:59.743981797 +0000 UTC m=+1197.608885844"
Sep 16 18:35:59.797826 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:35:59.797796    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:36:02.491931 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:02.491871    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:02.605142 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:02.605113    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:36:02.607775 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:02.607750    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:36:02.612328 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:02.612305    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:36:02.615135 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:02.615116    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:36:05.805495 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:05.805430    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:05.905441 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:05.905401    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:06.027403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.027373    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"]
Sep 16 18:36:06.027889 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.027873    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee0c731-debf-4fc9-afc2-8f8a54d279ca" containerName="istio-proxy"
Sep 16 18:36:06.030072 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.030055    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.033854 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.033829    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-8b875d\"/\"kube-root-ca.crt\""
Sep 16 18:36:06.033999 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.033827    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-8b875d\"/\"default-dockercfg-px777\""
Sep 16 18:36:06.033999 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.033827    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-8b875d\"/\"autores-model-8b875d-kserve-self-signed-certs\""
Sep 16 18:36:06.033999 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.033858    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-8b875d\"/\"openshift-service-ca.crt\""
Sep 16 18:36:06.044362 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.044335    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"]
Sep 16 18:36:06.063031 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.063004    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:06.131448 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.131416    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-home\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.131647 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.131466    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-tmp-dir\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.131647 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.131495    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7jnj\" (UniqueName: \"kubernetes.io/projected/6b93edfb-e344-4ac8-84ba-7378166856e7-kube-api-access-d7jnj\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.131647 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.131529    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-dshm\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.131647 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.131600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b93edfb-e344-4ac8-84ba-7378166856e7-tls-certs\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.131647 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.131621    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-model-cache\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.232681 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.232644    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-tmp-dir\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.232855 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.232692    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jnj\" (UniqueName: \"kubernetes.io/projected/6b93edfb-e344-4ac8-84ba-7378166856e7-kube-api-access-d7jnj\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.232855 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.232723    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-dshm\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.232855 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.232778    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b93edfb-e344-4ac8-84ba-7378166856e7-tls-certs\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.233001 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.232893    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-model-cache\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.233001 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.232972    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-home\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.233135 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.233101    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-tmp-dir\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.233275 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.233255    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-model-cache\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.233352 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.233334    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-home\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.235076 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.235057    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-dshm\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.235257 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.235239    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b93edfb-e344-4ac8-84ba-7378166856e7-tls-certs\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.242205 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.242182    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7jnj\" (UniqueName: \"kubernetes.io/projected/6b93edfb-e344-4ac8-84ba-7378166856e7-kube-api-access-d7jnj\") pod \"autores-model-8b875d-kserve-78b65977d7-p4kdr\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") " pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.340767 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.340670    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:06.623846 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.623814    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"]
Sep 16 18:36:06.624723 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:36:06.624653    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b93edfb_e344_4ac8_84ba_7378166856e7.slice/crio-b7c5e3298d5ef055217af56de8fc57089eedc48d5fd56b476a2ed445d3739d35 WatchSource:0}: Error finding container b7c5e3298d5ef055217af56de8fc57089eedc48d5fd56b476a2ed445d3739d35: Status 404 returned error can't find the container with id b7c5e3298d5ef055217af56de8fc57089eedc48d5fd56b476a2ed445d3739d35
Sep 16 18:36:06.828707 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.828666    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr" event={"ID":"6b93edfb-e344-4ac8-84ba-7378166856e7","Type":"ContainerStarted","Data":"18b513f2eb6b432be729dbc4e3283a034b4ddbb6e3e03948f428254c78cc4829"}
Sep 16 18:36:06.829145 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.828722    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr" event={"ID":"6b93edfb-e344-4ac8-84ba-7378166856e7","Type":"ContainerStarted","Data":"b7c5e3298d5ef055217af56de8fc57089eedc48d5fd56b476a2ed445d3739d35"}
Sep 16 18:36:06.830252 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.830229    2562 generic.go:356] "Generic (PLEG): container finished" podID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerID="f0db9a262d32efb570bc8c5907270d65a81f09441d671beb20ae8cf24ce9e43e" exitCode=0
Sep 16 18:36:06.830363 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.830305    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht" event={"ID":"6c8708e5-0029-4ffe-b4e1-65301c07efb8","Type":"ContainerDied","Data":"f0db9a262d32efb570bc8c5907270d65a81f09441d671beb20ae8cf24ce9e43e"}
Sep 16 18:36:06.830722 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.830706    2562 scope.go:117] "RemoveContainer" containerID="f0db9a262d32efb570bc8c5907270d65a81f09441d671beb20ae8cf24ce9e43e"
Sep 16 18:36:06.831842 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.831807    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:36:06.831916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.831881    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:36:06.831956 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.831914    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:36:06.853842 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:06.853794    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr" podStartSLOduration=0.853775606 podStartE2EDuration="853.775606ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:36:06.850036119 +0000 UTC m=+1204.714940170" watchObservedRunningTime="2026-09-16 18:36:06.853775606 +0000 UTC m=+1204.718679654"
Sep 16 18:36:07.838520 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:07.838488    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht" event={"ID":"6c8708e5-0029-4ffe-b4e1-65301c07efb8","Type":"ContainerStarted","Data":"d0a07590e35dda261f9007e9fd1e3a6348cd097c90c6e9821fa2b7bff65b9647"}
Sep 16 18:36:07.975114 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:07.975061    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:36:07.976363 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:07.976335    2562 patch_prober.go:28] interesting pod/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 18:36:07.976517 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:07.976385    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht" podUID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 18:36:08.610287 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:08.610250    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:36:08.975636 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:08.975598    2562 patch_prober.go:28] interesting pod/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 18:36:08.976226 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:08.976194    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht" podUID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 18:36:09.658477 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:09.658440    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:09.977383 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:09.977346    2562 patch_prober.go:28] interesting pod/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 18:36:09.978200 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:09.978094    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht" podUID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 18:36:10.975799 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:10.975747    2562 patch_prober.go:28] interesting pod/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 18:36:10.976021 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:10.975823    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht" podUID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 18:36:11.374190 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:11.374070    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:11.975860 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:11.975814    2562 patch_prober.go:28] interesting pod/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 18:36:11.976079 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:11.975893    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht" podUID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 18:36:12.376053 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:12.375942    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:12.702025 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:12.701972    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:36:12.975778 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:12.975694    2562 patch_prober.go:28] interesting pod/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 16 18:36:12.975942 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:12.975774    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht" podUID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 16 18:36:13.820813 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:13.820772    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:13.980458 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:13.980422    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:36:13.981520 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:13.981497    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:36:14.993503 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:14.993464    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:15.306744 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:15.306665    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:15.412393 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:15.412362    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:15.915060 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:15.915026    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:16.341888 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:16.341850    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:16.342396 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:16.341921    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:16.357530 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:16.357499    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:16.855797 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:16.855761    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:16.930667 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:16.930637    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:36:17.915733 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:17.915671    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:18.619282 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:18.619252    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:19.441601 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:19.441524    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:20.314284 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:20.314250    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:23.752072 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:23.752037    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"]
Sep 16 18:36:23.755446 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:23.755423    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:23.761526 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:23.761310    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4ab744\"/\"kube-root-ca.crt\""
Sep 16 18:36:23.763770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:23.763746    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4ab744\"/\"openshift-service-ca.crt\""
Sep 16 18:36:23.764024 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:23.764006    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4ab744\"/\"default-dockercfg-z4dk4\""
Sep 16 18:36:23.764643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:23.764272    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4ab744\"/\"test-model-4ab744-kserve-self-signed-certs\""
Sep 16 18:36:23.803122 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:23.803088    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:23.839889 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:23.839850    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"]
Sep 16 18:36:23.920853 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:23.920809    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-dshm\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:23.920853 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:23.920857    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-model-cache\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:23.921095 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:23.920879    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-home\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:23.921095 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:23.920907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-tmp-dir\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:23.921095 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:23.921004    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05c7d397-a627-4768-b2ef-42824aceae16-tls-certs\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:23.921095 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:23.921080    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfd6m\" (UniqueName: \"kubernetes.io/projected/05c7d397-a627-4768-b2ef-42824aceae16-kube-api-access-bfd6m\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:24.022627 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.022442    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfd6m\" (UniqueName: \"kubernetes.io/projected/05c7d397-a627-4768-b2ef-42824aceae16-kube-api-access-bfd6m\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:24.022627 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.022586    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-dshm\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:24.022874 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.022630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-model-cache\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:24.022874 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.022656    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-home\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:24.022874 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.022682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-tmp-dir\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:24.022874 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.022762    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05c7d397-a627-4768-b2ef-42824aceae16-tls-certs\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:24.023689 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.023660    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-home\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:24.023929 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.023907    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-tmp-dir\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:24.024782 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.024762    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-model-cache\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:24.026391 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.026370    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05c7d397-a627-4768-b2ef-42824aceae16-tls-certs\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:24.026746 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.026704    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-dshm\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:24.039309 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.039282    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfd6m\" (UniqueName: \"kubernetes.io/projected/05c7d397-a627-4768-b2ef-42824aceae16-kube-api-access-bfd6m\") pod \"test-model-4ab744-kserve-6d7689585-5rsvd\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") " pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:24.067622 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.067592    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:24.119006 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.118968    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:24.368237 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.368174    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"]
Sep 16 18:36:24.973434 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.973402    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd" event={"ID":"05c7d397-a627-4768-b2ef-42824aceae16","Type":"ContainerStarted","Data":"edf730514df422d015576b76ee162043d52d34fddb894241d7e0a23c54447977"}
Sep 16 18:36:24.973856 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:24.973445    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd" event={"ID":"05c7d397-a627-4768-b2ef-42824aceae16","Type":"ContainerStarted","Data":"4345701a35beb0b1b578d98c62b932a713adf326ce6ca7082155818879b19b11"}
Sep 16 18:36:25.000575 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:25.000511    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd" podStartSLOduration=2.000495859 podStartE2EDuration="2.000495859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:36:24.997207243 +0000 UTC m=+1222.862111289" watchObservedRunningTime="2026-09-16 18:36:25.000495859 +0000 UTC m=+1222.865399905"
Sep 16 18:36:25.106111 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:25.106075    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:25.318368 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:25.318284    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:25.448726 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:25.448691    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:25.650728 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:25.650629    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:25.812646 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:25.812611    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:26.046408 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:26.046373    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"]
Sep 16 18:36:26.046829 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:26.046632    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht" podUID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerName="istio-proxy" containerID="cri-o://d0a07590e35dda261f9007e9fd1e3a6348cd097c90c6e9821fa2b7bff65b9647" gracePeriod=30
Sep 16 18:36:26.062129 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:26.062095    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:26.532873 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:26.532842    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:26.638570 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:26.638527    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:27.071890 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:27.071848    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:27.516446 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:27.516410    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:28.130319 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:28.130287    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:28.983107 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:28.983066    2562 patch_prober.go:28] interesting pod/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 18:36:28.983283 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:28.983130    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht" podUID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:36:29.191004 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:29.190965    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:30.038008 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:30.037972    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:30.884803 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:30.884768    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:31.313030 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.313006    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:36:31.503616 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.503576    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpxp6\" (UniqueName: \"kubernetes.io/projected/6c8708e5-0029-4ffe-b4e1-65301c07efb8-kube-api-access-qpxp6\") pod \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") "
Sep 16 18:36:31.503791 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.503655    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-data\") pod \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") "
Sep 16 18:36:31.503791 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.503696    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-workload-certs\") pod \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") "
Sep 16 18:36:31.503791 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.503747    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-workload-socket\") pod \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") "
Sep 16 18:36:31.503791 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.503773    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istiod-ca-cert\") pod \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") "
Sep 16 18:36:31.504005 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.503817    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-podinfo\") pod \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") "
Sep 16 18:36:31.504005 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.503979    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-credential-socket\") pod \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") "
Sep 16 18:36:31.504111 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.503997    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c8708e5-0029-4ffe-b4e1-65301c07efb8" (UID: "6c8708e5-0029-4ffe-b4e1-65301c07efb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:31.504111 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.504045    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-envoy\") pod \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") "
Sep 16 18:36:31.504111 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.504092    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c8708e5-0029-4ffe-b4e1-65301c07efb8" (UID: "6c8708e5-0029-4ffe-b4e1-65301c07efb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:31.504272 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.504106    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c8708e5-0029-4ffe-b4e1-65301c07efb8" (UID: "6c8708e5-0029-4ffe-b4e1-65301c07efb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:31.504272 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.504116    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-token\") pod \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\" (UID: \"6c8708e5-0029-4ffe-b4e1-65301c07efb8\") "
Sep 16 18:36:31.504272 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.504149    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c8708e5-0029-4ffe-b4e1-65301c07efb8" (UID: "6c8708e5-0029-4ffe-b4e1-65301c07efb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:36:31.504272 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.504229    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c8708e5-0029-4ffe-b4e1-65301c07efb8" (UID: "6c8708e5-0029-4ffe-b4e1-65301c07efb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:31.504561 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.504527    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-workload-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:36:31.504643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.504573    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istiod-ca-cert\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:36:31.504643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.504582    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-credential-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:36:31.504643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.504593    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-data\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:36:31.504643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.504601    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-workload-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:36:31.506139 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.506111    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c8708e5-0029-4ffe-b4e1-65301c07efb8" (UID: "6c8708e5-0029-4ffe-b4e1-65301c07efb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:31.506264 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.506244    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c8708e5-0029-4ffe-b4e1-65301c07efb8" (UID: "6c8708e5-0029-4ffe-b4e1-65301c07efb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:36:31.506318 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.506302    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8708e5-0029-4ffe-b4e1-65301c07efb8-kube-api-access-qpxp6" (OuterVolumeSpecName: "kube-api-access-qpxp6") pod "6c8708e5-0029-4ffe-b4e1-65301c07efb8" (UID: "6c8708e5-0029-4ffe-b4e1-65301c07efb8"). InnerVolumeSpecName "kube-api-access-qpxp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:36:31.506381 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.506366    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c8708e5-0029-4ffe-b4e1-65301c07efb8" (UID: "6c8708e5-0029-4ffe-b4e1-65301c07efb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:36:31.605459 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.605431    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-podinfo\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:36:31.605459 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.605456    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-envoy\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:36:31.605459 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.605464    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8708e5-0029-4ffe-b4e1-65301c07efb8-istio-token\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:36:31.605729 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.605472    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpxp6\" (UniqueName: \"kubernetes.io/projected/6c8708e5-0029-4ffe-b4e1-65301c07efb8-kube-api-access-qpxp6\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:36:31.830614 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:31.830467    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:32.018743 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:32.018708    2562 generic.go:356] "Generic (PLEG): container finished" podID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerID="d0a07590e35dda261f9007e9fd1e3a6348cd097c90c6e9821fa2b7bff65b9647" exitCode=0
Sep 16 18:36:32.019164 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:32.018761    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"
Sep 16 18:36:32.019164 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:32.018788    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht" event={"ID":"6c8708e5-0029-4ffe-b4e1-65301c07efb8","Type":"ContainerDied","Data":"d0a07590e35dda261f9007e9fd1e3a6348cd097c90c6e9821fa2b7bff65b9647"}
Sep 16 18:36:32.019164 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:32.018850    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht" event={"ID":"6c8708e5-0029-4ffe-b4e1-65301c07efb8","Type":"ContainerDied","Data":"d3aec1b5e8dd6920a31430b924236f6b22e663e79400df17d8350d83a38be871"}
Sep 16 18:36:32.019164 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:32.018877    2562 scope.go:117] "RemoveContainer" containerID="d0a07590e35dda261f9007e9fd1e3a6348cd097c90c6e9821fa2b7bff65b9647"
Sep 16 18:36:32.029592 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:32.029532    2562 scope.go:117] "RemoveContainer" containerID="f0db9a262d32efb570bc8c5907270d65a81f09441d671beb20ae8cf24ce9e43e"
Sep 16 18:36:32.038125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:32.038109    2562 scope.go:117] "RemoveContainer" containerID="d0a07590e35dda261f9007e9fd1e3a6348cd097c90c6e9821fa2b7bff65b9647"
Sep 16 18:36:32.038391 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:36:32.038373    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0a07590e35dda261f9007e9fd1e3a6348cd097c90c6e9821fa2b7bff65b9647\": container with ID starting with d0a07590e35dda261f9007e9fd1e3a6348cd097c90c6e9821fa2b7bff65b9647 not found: ID does not exist" containerID="d0a07590e35dda261f9007e9fd1e3a6348cd097c90c6e9821fa2b7bff65b9647"
Sep 16 18:36:32.038429 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:32.038402    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0a07590e35dda261f9007e9fd1e3a6348cd097c90c6e9821fa2b7bff65b9647"} err="failed to get container status \"d0a07590e35dda261f9007e9fd1e3a6348cd097c90c6e9821fa2b7bff65b9647\": rpc error: code = NotFound desc = could not find container \"d0a07590e35dda261f9007e9fd1e3a6348cd097c90c6e9821fa2b7bff65b9647\": container with ID starting with d0a07590e35dda261f9007e9fd1e3a6348cd097c90c6e9821fa2b7bff65b9647 not found: ID does not exist"
Sep 16 18:36:32.038429 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:32.038419    2562 scope.go:117] "RemoveContainer" containerID="f0db9a262d32efb570bc8c5907270d65a81f09441d671beb20ae8cf24ce9e43e"
Sep 16 18:36:32.038693 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:36:32.038676    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0db9a262d32efb570bc8c5907270d65a81f09441d671beb20ae8cf24ce9e43e\": container with ID starting with f0db9a262d32efb570bc8c5907270d65a81f09441d671beb20ae8cf24ce9e43e not found: ID does not exist" containerID="f0db9a262d32efb570bc8c5907270d65a81f09441d671beb20ae8cf24ce9e43e"
Sep 16 18:36:32.038739 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:32.038699    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0db9a262d32efb570bc8c5907270d65a81f09441d671beb20ae8cf24ce9e43e"} err="failed to get container status \"f0db9a262d32efb570bc8c5907270d65a81f09441d671beb20ae8cf24ce9e43e\": rpc error: code = NotFound desc = could not find container \"f0db9a262d32efb570bc8c5907270d65a81f09441d671beb20ae8cf24ce9e43e\": container with ID starting with f0db9a262d32efb570bc8c5907270d65a81f09441d671beb20ae8cf24ce9e43e not found: ID does not exist"
Sep 16 18:36:32.043510 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:32.043487    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"]
Sep 16 18:36:32.048028 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:32.047995    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ad2bfa56-gw-openshift-default-85f877fd88-lzhht"]
Sep 16 18:36:32.552240 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:32.552201    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" path="/var/lib/kubelet/pods/6c8708e5-0029-4ffe-b4e1-65301c07efb8/volumes"
Sep 16 18:36:34.068725 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:34.068691    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:34.069128 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:34.068765    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:34.081171 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:34.081143    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:35.038321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:35.038289    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:36:35.048941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:35.048910    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:36:39.304050 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:39.304010    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:40.358932 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:40.358892    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:41.322983 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:41.322929    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:42.157539 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:42.157496    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:42.285117 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:42.285086    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:42.413737 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:42.413633    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:43.702478 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:43.702440    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:44.554568 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:44.554516    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:45.523486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:45.523448    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:46.658828 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:46.658789    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:46.753837 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:46.753800    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:46.899786 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:46.899753    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:47.382334 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:47.382301    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:47.504417 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:47.504384    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:48.044018 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:48.043987    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:49.100629 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:49.100598    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:49.868348 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:49.868317    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:50.537785 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:50.537753    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:50.635297 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:50.635263    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:50.743387 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:50.743356    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:50.853332 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:50.853235    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:50.953322 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:50.953275    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:51.266738 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:51.266704    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:51.397199 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:51.397170    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:52.165733 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:52.165697    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:53.207748 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:53.207689    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:54.115517 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:54.115485    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:55.023430 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:55.023396    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:55.657488 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:55.657447    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:36:55.942828 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:55.942736    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:56.377382 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:56.377348    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"]
Sep 16 18:36:56.456376 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:36:56.456339    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:00.426772 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:00.426712    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:07.992050 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:07.992018    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:09.560094 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:09.560054    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:10.276925 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:10.276887    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:11.073124 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.073093    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:11.396846 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.396758    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr" podUID="6b93edfb-e344-4ac8-84ba-7378166856e7" containerName="main" containerID="cri-o://18b513f2eb6b432be729dbc4e3283a034b4ddbb6e3e03948f428254c78cc4829" gracePeriod=45
Sep 16 18:37:11.647489 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.647425    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:37:11.800445 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.800408    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-dshm\") pod \"6b93edfb-e344-4ac8-84ba-7378166856e7\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") "
Sep 16 18:37:11.800646 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.800479    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-model-cache\") pod \"6b93edfb-e344-4ac8-84ba-7378166856e7\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") "
Sep 16 18:37:11.800646 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.800521    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-tmp-dir\") pod \"6b93edfb-e344-4ac8-84ba-7378166856e7\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") "
Sep 16 18:37:11.800646 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.800577    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7jnj\" (UniqueName: \"kubernetes.io/projected/6b93edfb-e344-4ac8-84ba-7378166856e7-kube-api-access-d7jnj\") pod \"6b93edfb-e344-4ac8-84ba-7378166856e7\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") "
Sep 16 18:37:11.800773 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.800655    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b93edfb-e344-4ac8-84ba-7378166856e7-tls-certs\") pod \"6b93edfb-e344-4ac8-84ba-7378166856e7\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") "
Sep 16 18:37:11.800773 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.800680    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-home\") pod \"6b93edfb-e344-4ac8-84ba-7378166856e7\" (UID: \"6b93edfb-e344-4ac8-84ba-7378166856e7\") "
Sep 16 18:37:11.800869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.800814    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b93edfb-e344-4ac8-84ba-7378166856e7" (UID: "6b93edfb-e344-4ac8-84ba-7378166856e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:11.800869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.800825    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b93edfb-e344-4ac8-84ba-7378166856e7" (UID: "6b93edfb-e344-4ac8-84ba-7378166856e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:11.801021 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.801000    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-home" (OuterVolumeSpecName: "home") pod "6b93edfb-e344-4ac8-84ba-7378166856e7" (UID: "6b93edfb-e344-4ac8-84ba-7378166856e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:11.801099 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.801076    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:11.801140 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.801105    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:11.802866 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.802848    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-dshm" (OuterVolumeSpecName: "dshm") pod "6b93edfb-e344-4ac8-84ba-7378166856e7" (UID: "6b93edfb-e344-4ac8-84ba-7378166856e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:11.802968 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.802951    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b93edfb-e344-4ac8-84ba-7378166856e7-kube-api-access-d7jnj" (OuterVolumeSpecName: "kube-api-access-d7jnj") pod "6b93edfb-e344-4ac8-84ba-7378166856e7" (UID: "6b93edfb-e344-4ac8-84ba-7378166856e7"). InnerVolumeSpecName "kube-api-access-d7jnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:37:11.803005 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.802976    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b93edfb-e344-4ac8-84ba-7378166856e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b93edfb-e344-4ac8-84ba-7378166856e7" (UID: "6b93edfb-e344-4ac8-84ba-7378166856e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:37:11.902283 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.902191    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b93edfb-e344-4ac8-84ba-7378166856e7-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:11.902283 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.902226    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:11.902283 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.902235    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b93edfb-e344-4ac8-84ba-7378166856e7-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:11.902283 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:11.902243    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7jnj\" (UniqueName: \"kubernetes.io/projected/6b93edfb-e344-4ac8-84ba-7378166856e7-kube-api-access-d7jnj\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:12.050900 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:12.050853    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:12.281320 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:12.281284    2562 generic.go:356] "Generic (PLEG): container finished" podID="6b93edfb-e344-4ac8-84ba-7378166856e7" containerID="18b513f2eb6b432be729dbc4e3283a034b4ddbb6e3e03948f428254c78cc4829" exitCode=0
Sep 16 18:37:12.281320 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:12.281323    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"
Sep 16 18:37:12.281958 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:12.281364    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr" event={"ID":"6b93edfb-e344-4ac8-84ba-7378166856e7","Type":"ContainerDied","Data":"18b513f2eb6b432be729dbc4e3283a034b4ddbb6e3e03948f428254c78cc4829"}
Sep 16 18:37:12.281958 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:12.281404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr" event={"ID":"6b93edfb-e344-4ac8-84ba-7378166856e7","Type":"ContainerDied","Data":"b7c5e3298d5ef055217af56de8fc57089eedc48d5fd56b476a2ed445d3739d35"}
Sep 16 18:37:12.281958 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:12.281420    2562 scope.go:117] "RemoveContainer" containerID="18b513f2eb6b432be729dbc4e3283a034b4ddbb6e3e03948f428254c78cc4829"
Sep 16 18:37:12.293975 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:12.293951    2562 scope.go:117] "RemoveContainer" containerID="18b513f2eb6b432be729dbc4e3283a034b4ddbb6e3e03948f428254c78cc4829"
Sep 16 18:37:12.294299 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:37:12.294280    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b513f2eb6b432be729dbc4e3283a034b4ddbb6e3e03948f428254c78cc4829\": container with ID starting with 18b513f2eb6b432be729dbc4e3283a034b4ddbb6e3e03948f428254c78cc4829 not found: ID does not exist" containerID="18b513f2eb6b432be729dbc4e3283a034b4ddbb6e3e03948f428254c78cc4829"
Sep 16 18:37:12.294374 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:12.294307    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b513f2eb6b432be729dbc4e3283a034b4ddbb6e3e03948f428254c78cc4829"} err="failed to get container status \"18b513f2eb6b432be729dbc4e3283a034b4ddbb6e3e03948f428254c78cc4829\": rpc error: code = NotFound desc = could not find container \"18b513f2eb6b432be729dbc4e3283a034b4ddbb6e3e03948f428254c78cc4829\": container with ID starting with 18b513f2eb6b432be729dbc4e3283a034b4ddbb6e3e03948f428254c78cc4829 not found: ID does not exist"
Sep 16 18:37:12.310211 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:12.310183    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"]
Sep 16 18:37:12.314938 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:12.314914    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-8b875d/autores-model-8b875d-kserve-78b65977d7-p4kdr"]
Sep 16 18:37:12.552387 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:12.552313    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b93edfb-e344-4ac8-84ba-7378166856e7" path="/var/lib/kubelet/pods/6b93edfb-e344-4ac8-84ba-7378166856e7/volumes"
Sep 16 18:37:12.904031 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:12.903944    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:13.021694 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.021662    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw"]
Sep 16 18:37:13.022080 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.022068    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerName="istio-proxy"
Sep 16 18:37:13.022124 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.022082    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerName="istio-proxy"
Sep 16 18:37:13.022124 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.022100    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerName="istio-proxy"
Sep 16 18:37:13.022124 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.022105    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerName="istio-proxy"
Sep 16 18:37:13.022124 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.022114    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b93edfb-e344-4ac8-84ba-7378166856e7" containerName="main"
Sep 16 18:37:13.022124 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.022119    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b93edfb-e344-4ac8-84ba-7378166856e7" containerName="main"
Sep 16 18:37:13.022274 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.022179    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerName="istio-proxy"
Sep 16 18:37:13.022274 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.022188    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b93edfb-e344-4ac8-84ba-7378166856e7" containerName="main"
Sep 16 18:37:13.022274 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.022195    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c8708e5-0029-4ffe-b4e1-65301c07efb8" containerName="istio-proxy"
Sep 16 18:37:13.024980 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.024961    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw"
Sep 16 18:37:13.027980 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.027956    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:37:13.028183 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.028170    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:37:13.035926 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.035903    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw"]
Sep 16 18:37:13.219016 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.218980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a68a61-9939-4122-982e-bb930e89f3fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw\" (UID: \"61a68a61-9939-4122-982e-bb930e89f3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw"
Sep 16 18:37:13.319893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.319844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a68a61-9939-4122-982e-bb930e89f3fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw\" (UID: \"61a68a61-9939-4122-982e-bb930e89f3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw"
Sep 16 18:37:13.320433 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.320404    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a68a61-9939-4122-982e-bb930e89f3fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw\" (UID: \"61a68a61-9939-4122-982e-bb930e89f3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw"
Sep 16 18:37:13.335502 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.335475    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw"
Sep 16 18:37:13.611174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.611144    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw"]
Sep 16 18:37:13.611501 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:37:13.611464    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a68a61_9939_4122_982e_bb930e89f3fc.slice/crio-7fa36b054bc5bb20445c553bbb35c6b537b2df9fe0e3d04812ca87619d2e4749 WatchSource:0}: Error finding container 7fa36b054bc5bb20445c553bbb35c6b537b2df9fe0e3d04812ca87619d2e4749: Status 404 returned error can't find the container with id 7fa36b054bc5bb20445c553bbb35c6b537b2df9fe0e3d04812ca87619d2e4749
Sep 16 18:37:13.740562 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:13.740507    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:14.296884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:14.296844    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw" event={"ID":"61a68a61-9939-4122-982e-bb930e89f3fc","Type":"ContainerStarted","Data":"c3b63222922f9db9004e969efa3235e59928e1f0fb744f30d82369cf72c1a13d"}
Sep 16 18:37:14.297053 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:14.296907    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw" event={"ID":"61a68a61-9939-4122-982e-bb930e89f3fc","Type":"ContainerStarted","Data":"7fa36b054bc5bb20445c553bbb35c6b537b2df9fe0e3d04812ca87619d2e4749"}
Sep 16 18:37:14.315319 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:14.315272    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw" podStartSLOduration=1.315255931 podStartE2EDuration="1.315255931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:37:14.314484761 +0000 UTC m=+1272.179388809" watchObservedRunningTime="2026-09-16 18:37:14.315255931 +0000 UTC m=+1272.180159977"
Sep 16 18:37:14.654814 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:14.654726    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:15.304454 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:15.304419    2562 generic.go:356] "Generic (PLEG): container finished" podID="61a68a61-9939-4122-982e-bb930e89f3fc" containerID="c3b63222922f9db9004e969efa3235e59928e1f0fb744f30d82369cf72c1a13d" exitCode=0
Sep 16 18:37:15.304637 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:15.304462    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw" event={"ID":"61a68a61-9939-4122-982e-bb930e89f3fc","Type":"ContainerDied","Data":"c3b63222922f9db9004e969efa3235e59928e1f0fb744f30d82369cf72c1a13d"}
Sep 16 18:37:15.447034 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:15.446996    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:16.443339 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:16.443313    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw"
Sep 16 18:37:16.550055 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:16.550017    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a68a61-9939-4122-982e-bb930e89f3fc-maas-api-service-ca\") pod \"61a68a61-9939-4122-982e-bb930e89f3fc\" (UID: \"61a68a61-9939-4122-982e-bb930e89f3fc\") "
Sep 16 18:37:16.550535 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:16.550486    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a68a61-9939-4122-982e-bb930e89f3fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "61a68a61-9939-4122-982e-bb930e89f3fc" (UID: "61a68a61-9939-4122-982e-bb930e89f3fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:37:16.550769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:16.550747    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61a68a61-9939-4122-982e-bb930e89f3fc-maas-api-service-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:16.619710 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:16.619674    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:17.016674 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.016642    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"]
Sep 16 18:37:17.017102 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.017087    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a68a61-9939-4122-982e-bb930e89f3fc" containerName="revoke-keys"
Sep 16 18:37:17.017102 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.017105    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a68a61-9939-4122-982e-bb930e89f3fc" containerName="revoke-keys"
Sep 16 18:37:17.017196 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.017188    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="61a68a61-9939-4122-982e-bb930e89f3fc" containerName="revoke-keys"
Sep 16 18:37:17.019390 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.019374    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.022378 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.022352    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-909a40\"/\"premium-w0-909a40-kserve-self-signed-certs\""
Sep 16 18:37:17.036813 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.036787    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"]
Sep 16 18:37:17.105767 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.105724    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:17.158722 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.158215    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-home\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.158722 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.158287    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-dshm\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.158722 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.158390    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-tmp-dir\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.158722 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.158477    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l9vm\" (UniqueName: \"kubernetes.io/projected/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-kube-api-access-2l9vm\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.158722 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.158522    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-model-cache\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.158722 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.158567    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-tls-certs\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.259240 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.259206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9vm\" (UniqueName: \"kubernetes.io/projected/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-kube-api-access-2l9vm\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.259412 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.259257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-model-cache\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.259412 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.259288    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-tls-certs\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.259412 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.259337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-home\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.259412 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.259377    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-dshm\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.259696 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.259436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-tmp-dir\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.259783 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.259763    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-model-cache\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.259853 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.259793    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-home\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.259914 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.259892    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-tmp-dir\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.261779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.261746    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-dshm\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.262024 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.262009    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-tls-certs\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.268455 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.268391    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l9vm\" (UniqueName: \"kubernetes.io/projected/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-kube-api-access-2l9vm\") pod \"premium-w0-909a40-kserve-5c49f47ffc-zk8pb\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") " pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.318529 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.318502    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw"
Sep 16 18:37:17.318529 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.318523    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw" event={"ID":"61a68a61-9939-4122-982e-bb930e89f3fc","Type":"ContainerDied","Data":"7fa36b054bc5bb20445c553bbb35c6b537b2df9fe0e3d04812ca87619d2e4749"}
Sep 16 18:37:17.318786 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.318584    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fa36b054bc5bb20445c553bbb35c6b537b2df9fe0e3d04812ca87619d2e4749"
Sep 16 18:37:17.329413 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.329382    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:17.491689 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.491654    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:17.608359 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:17.608333    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"]
Sep 16 18:37:17.609313 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:37:17.609283    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0fe54a1_f24e_479d_b114_fb8bc7dd370f.slice/crio-c0ed5b230cf53ee1b19987521088f57317ec9ecdfa1a3bd94d922fc3ce383af4 WatchSource:0}: Error finding container c0ed5b230cf53ee1b19987521088f57317ec9ecdfa1a3bd94d922fc3ce383af4: Status 404 returned error can't find the container with id c0ed5b230cf53ee1b19987521088f57317ec9ecdfa1a3bd94d922fc3ce383af4
Sep 16 18:37:18.234958 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:18.234918    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:18.330670 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:18.330638    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb" event={"ID":"b0fe54a1-f24e-479d-b114-fb8bc7dd370f","Type":"ContainerStarted","Data":"1034699853ad34a72cf946b69c221fff79e3da8bd0f9230dc9a6dc4bb72402d1"}
Sep 16 18:37:18.330840 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:18.330679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb" event={"ID":"b0fe54a1-f24e-479d-b114-fb8bc7dd370f","Type":"ContainerStarted","Data":"c0ed5b230cf53ee1b19987521088f57317ec9ecdfa1a3bd94d922fc3ce383af4"}
Sep 16 18:37:18.352186 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:18.352125    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb" podStartSLOduration=2.35210862 podStartE2EDuration="2.35210862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:37:18.348984004 +0000 UTC m=+1276.213888052" watchObservedRunningTime="2026-09-16 18:37:18.35210862 +0000 UTC m=+1276.217012673"
Sep 16 18:37:19.149590 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:19.149487    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:19.375331 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:19.375286    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"]
Sep 16 18:37:19.375624 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:19.375590    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp" podUID="2e520ad7-28d2-4a42-8bf3-30e3159a53f9" containerName="istio-proxy" containerID="cri-o://4359c963fa6c405a8a5c0c46b6f0a62086f580f4e61b972866d8b535a902e86c" gracePeriod=30
Sep 16 18:37:19.406583 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:19.406496    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:19.841728 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:19.841694    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:20.181232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:20.181141    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:20.430460 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:20.430425    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:22.930133 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:22.930099    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:23.026342 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:23.026304    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:23.897750 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:23.897715    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:24.633871 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.633848    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:37:24.727581 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.727507    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istiod-ca-cert\") pod \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") "
Sep 16 18:37:24.727779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.727588    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-workload-certs\") pod \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") "
Sep 16 18:37:24.727779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.727614    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-envoy\") pod \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") "
Sep 16 18:37:24.727779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.727656    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-podinfo\") pod \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") "
Sep 16 18:37:24.727779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.727727    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-data\") pod \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") "
Sep 16 18:37:24.727779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.727752    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-workload-socket\") pod \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") "
Sep 16 18:37:24.728044 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.727787    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-credential-socket\") pod \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") "
Sep 16 18:37:24.728044 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.727815    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-token\") pod \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") "
Sep 16 18:37:24.728044 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.727838    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q9f4\" (UniqueName: \"kubernetes.io/projected/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-kube-api-access-9q9f4\") pod \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\" (UID: \"2e520ad7-28d2-4a42-8bf3-30e3159a53f9\") "
Sep 16 18:37:24.728201 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.728055    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2e520ad7-28d2-4a42-8bf3-30e3159a53f9" (UID: "2e520ad7-28d2-4a42-8bf3-30e3159a53f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:24.728201 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.727978    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2e520ad7-28d2-4a42-8bf3-30e3159a53f9" (UID: "2e520ad7-28d2-4a42-8bf3-30e3159a53f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:37:24.728201 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.728099    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2e520ad7-28d2-4a42-8bf3-30e3159a53f9" (UID: "2e520ad7-28d2-4a42-8bf3-30e3159a53f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:24.728201 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.728164    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2e520ad7-28d2-4a42-8bf3-30e3159a53f9" (UID: "2e520ad7-28d2-4a42-8bf3-30e3159a53f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:24.728389 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.728346    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-data" (OuterVolumeSpecName: "istio-data") pod "2e520ad7-28d2-4a42-8bf3-30e3159a53f9" (UID: "2e520ad7-28d2-4a42-8bf3-30e3159a53f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:24.728389 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.728365    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-workload-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:24.728389 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.728386    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-credential-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:24.728565 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.728400    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istiod-ca-cert\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:24.728565 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.728414    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-workload-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:24.729953 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.729922    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2e520ad7-28d2-4a42-8bf3-30e3159a53f9" (UID: "2e520ad7-28d2-4a42-8bf3-30e3159a53f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:24.730149 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.730125    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-kube-api-access-9q9f4" (OuterVolumeSpecName: "kube-api-access-9q9f4") pod "2e520ad7-28d2-4a42-8bf3-30e3159a53f9" (UID: "2e520ad7-28d2-4a42-8bf3-30e3159a53f9"). InnerVolumeSpecName "kube-api-access-9q9f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:37:24.730232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.730143    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "2e520ad7-28d2-4a42-8bf3-30e3159a53f9" (UID: "2e520ad7-28d2-4a42-8bf3-30e3159a53f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:37:24.730232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.730144    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2e520ad7-28d2-4a42-8bf3-30e3159a53f9" (UID: "2e520ad7-28d2-4a42-8bf3-30e3159a53f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:37:24.759864 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.759838    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:24.829983 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.829945    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-envoy\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:24.829983 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.829976    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-podinfo\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:24.829983 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.829985    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-data\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:24.829983 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.829992    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-istio-token\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:24.830246 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.830000    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q9f4\" (UniqueName: \"kubernetes.io/projected/2e520ad7-28d2-4a42-8bf3-30e3159a53f9-kube-api-access-9q9f4\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:37:24.949591 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:24.949523    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:37:25.376833 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:25.376802    2562 generic.go:356] "Generic (PLEG): container finished" podID="2e520ad7-28d2-4a42-8bf3-30e3159a53f9" containerID="4359c963fa6c405a8a5c0c46b6f0a62086f580f4e61b972866d8b535a902e86c" exitCode=0
Sep 16 18:37:25.377047 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:25.376868    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"
Sep 16 18:37:25.377047 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:25.376894    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp" event={"ID":"2e520ad7-28d2-4a42-8bf3-30e3159a53f9","Type":"ContainerDied","Data":"4359c963fa6c405a8a5c0c46b6f0a62086f580f4e61b972866d8b535a902e86c"}
Sep 16 18:37:25.377047 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:25.376965    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp" event={"ID":"2e520ad7-28d2-4a42-8bf3-30e3159a53f9","Type":"ContainerDied","Data":"cbd5c748751ae42e399e3f59f7f01e3c936c21e7b7de124c0b54794213c3c003"}
Sep 16 18:37:25.377047 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:25.377000    2562 scope.go:117] "RemoveContainer" containerID="4359c963fa6c405a8a5c0c46b6f0a62086f580f4e61b972866d8b535a902e86c"
Sep 16 18:37:25.388290 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:25.388271    2562 scope.go:117] "RemoveContainer" containerID="85858c37c4e91192fb10fe9ecc8d21b23675b2840d8ae50a2a9b1db2818c61c5"
Sep 16 18:37:25.399148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:25.399127    2562 scope.go:117] "RemoveContainer" containerID="4359c963fa6c405a8a5c0c46b6f0a62086f580f4e61b972866d8b535a902e86c"
Sep 16 18:37:25.399452 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:37:25.399431    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4359c963fa6c405a8a5c0c46b6f0a62086f580f4e61b972866d8b535a902e86c\": container with ID starting with 4359c963fa6c405a8a5c0c46b6f0a62086f580f4e61b972866d8b535a902e86c not found: ID does not exist" containerID="4359c963fa6c405a8a5c0c46b6f0a62086f580f4e61b972866d8b535a902e86c"
Sep 16 18:37:25.399527 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:25.399465    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4359c963fa6c405a8a5c0c46b6f0a62086f580f4e61b972866d8b535a902e86c"} err="failed to get container status \"4359c963fa6c405a8a5c0c46b6f0a62086f580f4e61b972866d8b535a902e86c\": rpc error: code = NotFound desc = could not find container \"4359c963fa6c405a8a5c0c46b6f0a62086f580f4e61b972866d8b535a902e86c\": container with ID starting with 4359c963fa6c405a8a5c0c46b6f0a62086f580f4e61b972866d8b535a902e86c not found: ID does not exist"
Sep 16 18:37:25.399527 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:25.399489    2562 scope.go:117] "RemoveContainer" containerID="85858c37c4e91192fb10fe9ecc8d21b23675b2840d8ae50a2a9b1db2818c61c5"
Sep 16 18:37:25.399793 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:37:25.399774    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85858c37c4e91192fb10fe9ecc8d21b23675b2840d8ae50a2a9b1db2818c61c5\": container with ID starting with 85858c37c4e91192fb10fe9ecc8d21b23675b2840d8ae50a2a9b1db2818c61c5 not found: ID does not exist" containerID="85858c37c4e91192fb10fe9ecc8d21b23675b2840d8ae50a2a9b1db2818c61c5"
Sep 16 18:37:25.399832 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:25.399799    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85858c37c4e91192fb10fe9ecc8d21b23675b2840d8ae50a2a9b1db2818c61c5"} err="failed to get container status \"85858c37c4e91192fb10fe9ecc8d21b23675b2840d8ae50a2a9b1db2818c61c5\": rpc error: code = NotFound desc = could not find container \"85858c37c4e91192fb10fe9ecc8d21b23675b2840d8ae50a2a9b1db2818c61c5\": container with ID starting with 85858c37c4e91192fb10fe9ecc8d21b23675b2840d8ae50a2a9b1db2818c61c5 not found: ID does not exist"
Sep 16 18:37:25.414818 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:25.414776    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"]
Sep 16 18:37:25.417430 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:25.417407    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-8b875d-openshift-default-75c4bf76d6-2ddkp"]
Sep 16 18:37:26.033005 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:26.032951    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:26.551622 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:26.551585    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e520ad7-28d2-4a42-8bf3-30e3159a53f9" path="/var/lib/kubelet/pods/2e520ad7-28d2-4a42-8bf3-30e3159a53f9/volumes"
Sep 16 18:37:26.948390 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:26.948307    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:27.116830 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:27.116797    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:27.273707 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:27.273674    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:27.329842 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:27.329805    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:27.330077 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:27.330063    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:27.352846 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:27.352815    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:27.402041 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:27.402016    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:37:27.504809 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:27.504772    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw"]
Sep 16 18:37:27.513523 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:27.513492    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-019b7d43-p2xkw"]
Sep 16 18:37:27.514001 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:27.513974    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:27.655942 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:27.655845    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:27.785908 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:27.785872    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:28.115807 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:28.115776    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:28.245218 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:28.245186    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:28.356649 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:28.356615    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:28.552319 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:28.552283    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61a68a61-9939-4122-982e-bb930e89f3fc" path="/var/lib/kubelet/pods/61a68a61-9939-4122-982e-bb930e89f3fc/volumes"
Sep 16 18:37:28.765917 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:28.765888    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:28.908210 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:28.908131    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:29.237887 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.237854    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:29.522737 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.522648    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"]
Sep 16 18:37:29.523117 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.523102    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e520ad7-28d2-4a42-8bf3-30e3159a53f9" containerName="istio-proxy"
Sep 16 18:37:29.523160 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.523118    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e520ad7-28d2-4a42-8bf3-30e3159a53f9" containerName="istio-proxy"
Sep 16 18:37:29.523160 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.523135    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e520ad7-28d2-4a42-8bf3-30e3159a53f9" containerName="istio-proxy"
Sep 16 18:37:29.523160 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.523140    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e520ad7-28d2-4a42-8bf3-30e3159a53f9" containerName="istio-proxy"
Sep 16 18:37:29.523258 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.523209    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e520ad7-28d2-4a42-8bf3-30e3159a53f9" containerName="istio-proxy"
Sep 16 18:37:29.526290 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.526272    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.529181 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.529160    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b95e3be6-openshift-default-dockercfg-pghsz\""
Sep 16 18:37:29.545304 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.545279    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"]
Sep 16 18:37:29.568686 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.568655    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:29.576902 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.576874    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-credential-socket\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.576994 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.576934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-workload-socket\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.577040 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.576996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqhnq\" (UniqueName: \"kubernetes.io/projected/39d27bcc-faf3-463c-8818-ac52063d2831-kube-api-access-wqhnq\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.577183 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.577171    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39d27bcc-faf3-463c-8818-ac52063d2831-istio-podinfo\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.577234 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.577197    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-workload-certs\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.577388 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.577377    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39d27bcc-faf3-463c-8818-ac52063d2831-istio-token\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.577421 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.577407    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-istio-envoy\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.577802 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.577783    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39d27bcc-faf3-463c-8818-ac52063d2831-istiod-ca-cert\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.577912 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.577832    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-istio-data\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.679172 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.679135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-credential-socket\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.679172 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.679180    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-workload-socket\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.679370 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.679302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqhnq\" (UniqueName: \"kubernetes.io/projected/39d27bcc-faf3-463c-8818-ac52063d2831-kube-api-access-wqhnq\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.679370 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.679348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39d27bcc-faf3-463c-8818-ac52063d2831-istio-podinfo\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.679479 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.679384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-workload-certs\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.679479 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.679438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39d27bcc-faf3-463c-8818-ac52063d2831-istio-token\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.679612 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.679483    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-istio-envoy\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.679612 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.679570    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-workload-socket\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.679721 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.679620    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-credential-socket\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.679721 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.679625    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39d27bcc-faf3-463c-8818-ac52063d2831-istiod-ca-cert\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.679721 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.679688    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-workload-certs\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.679721 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.679693    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-istio-data\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.679966 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.679941    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-istio-data\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.680179 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.680161    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39d27bcc-faf3-463c-8818-ac52063d2831-istiod-ca-cert\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.681696 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.681666    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-istio-envoy\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.681869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.681848    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39d27bcc-faf3-463c-8818-ac52063d2831-istio-podinfo\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.689341 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.689312    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39d27bcc-faf3-463c-8818-ac52063d2831-istio-token\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.689516 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.689497    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqhnq\" (UniqueName: \"kubernetes.io/projected/39d27bcc-faf3-463c-8818-ac52063d2831-kube-api-access-wqhnq\") pod \"e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") " pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.836174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.836093    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:29.891868 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.891774    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:29.990604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:29.990570    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:30.136814 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:30.136780    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"]
Sep 16 18:37:30.142955 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:30.141218    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:37:30.142955 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:30.141321    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:37:30.142955 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:30.141365    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:37:30.412937 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:30.412840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst" event={"ID":"39d27bcc-faf3-463c-8818-ac52063d2831","Type":"ContainerStarted","Data":"336ac8ec429c3ed1f0551b59621179c55dd1658878a7bd042fc23c3a7480b7e8"}
Sep 16 18:37:30.412937 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:30.412904    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst" event={"ID":"39d27bcc-faf3-463c-8818-ac52063d2831","Type":"ContainerStarted","Data":"de8b08679b226083ab5be29c103a3ce159a33636c99e388f4cde1d6877f9a2a3"}
Sep 16 18:37:30.442018 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:30.441966    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst" podStartSLOduration=1.4419509879999999 podStartE2EDuration="1.441950988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:37:30.439105919 +0000 UTC m=+1288.304009964" watchObservedRunningTime="2026-09-16 18:37:30.441950988 +0000 UTC m=+1288.306855034"
Sep 16 18:37:30.836886 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:30.836854    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:31.198816 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:31.198789    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:31.841645 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:31.841606    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:31.930818 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:31.930784    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:32.073207 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:32.073176    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:32.301347 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:32.301318    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:32.424810 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:32.424778    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:32.425819 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:32.425798    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:37:33.419420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:33.419387    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:34.162427 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:34.162394    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:35.005255 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:35.005226    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:35.118891 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:35.118859    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:35.378737 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:35.378649    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:35.630742 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:35.630668    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:35.768552 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:35.768504    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:36.230653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:36.230623    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:37.337360 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:37.337281    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:38.309577 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:38.309524    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:39.718946 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:39.718909    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:48.696298 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:48.696268    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:50.168833 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:50.168787    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:51.071625 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:51.071583    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:51.799441 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:51.799408    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:52.124129 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:52.124053    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:37:52.737203 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:52.737172    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:53.052641 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:53.052530    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:53.544119 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:53.544084    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:54.417500 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:54.417465    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:55.360699 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:55.360666    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:56.078788 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:56.078748    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:56.648335 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:56.648303    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v"]
Sep 16 18:37:56.648966 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:56.648935    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e520ad7-28d2-4a42-8bf3-30e3159a53f9" containerName="istio-proxy"
Sep 16 18:37:56.656910 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:56.656876    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v"
Sep 16 18:37:56.659595 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:56.659572    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:37:56.660106 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:56.659917    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:37:56.661020 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:56.660989    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v"]
Sep 16 18:37:56.665516 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:56.665478    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7858edd0-1e22-469e-b1e5-45a5bb08df03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v\" (UID: \"7858edd0-1e22-469e-b1e5-45a5bb08df03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v"
Sep 16 18:37:56.767567 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:56.766523    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7858edd0-1e22-469e-b1e5-45a5bb08df03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v\" (UID: \"7858edd0-1e22-469e-b1e5-45a5bb08df03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v"
Sep 16 18:37:56.767567 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:56.767388    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7858edd0-1e22-469e-b1e5-45a5bb08df03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v\" (UID: \"7858edd0-1e22-469e-b1e5-45a5bb08df03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v"
Sep 16 18:37:56.934598 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:56.934457    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:56.973675 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:56.973641    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v"
Sep 16 18:37:57.263136 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:57.261100    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v"]
Sep 16 18:37:57.266456 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:37:57.266413    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7858edd0_1e22_469e_b1e5_45a5bb08df03.slice/crio-82809d2b1cef96a8ed7c3e572c88229266586c7940334262d9b43d0c99804143 WatchSource:0}: Error finding container 82809d2b1cef96a8ed7c3e572c88229266586c7940334262d9b43d0c99804143: Status 404 returned error can't find the container with id 82809d2b1cef96a8ed7c3e572c88229266586c7940334262d9b43d0c99804143
Sep 16 18:37:57.589223 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:57.589186    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v" event={"ID":"7858edd0-1e22-469e-b1e5-45a5bb08df03","Type":"ContainerStarted","Data":"82809d2b1cef96a8ed7c3e572c88229266586c7940334262d9b43d0c99804143"}
Sep 16 18:37:57.693484 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:57.693447    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:58.556623 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:58.554419    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:58.596974 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:58.596942    2562 generic.go:356] "Generic (PLEG): container finished" podID="7858edd0-1e22-469e-b1e5-45a5bb08df03" containerID="beafc65a716c1d4e1cdee0d1fc6208b3a84ed15cb39b9f35227b9e832ebf302a" exitCode=0
Sep 16 18:37:58.597123 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:58.596991    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v" event={"ID":"7858edd0-1e22-469e-b1e5-45a5bb08df03","Type":"ContainerDied","Data":"beafc65a716c1d4e1cdee0d1fc6208b3a84ed15cb39b9f35227b9e832ebf302a"}
Sep 16 18:37:59.363109 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:59.363074    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:59.766433 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:59.766404    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v"
Sep 16 18:37:59.796516 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:59.796482    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7858edd0-1e22-469e-b1e5-45a5bb08df03-maas-api-service-ca\") pod \"7858edd0-1e22-469e-b1e5-45a5bb08df03\" (UID: \"7858edd0-1e22-469e-b1e5-45a5bb08df03\") "
Sep 16 18:37:59.797435 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:59.797401    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7858edd0-1e22-469e-b1e5-45a5bb08df03-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7858edd0-1e22-469e-b1e5-45a5bb08df03" (UID: "7858edd0-1e22-469e-b1e5-45a5bb08df03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:37:59.898670 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:37:59.898583    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7858edd0-1e22-469e-b1e5-45a5bb08df03-maas-api-service-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:38:00.168236 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:00.168159    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:00.611058 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:00.611026    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v"
Sep 16 18:38:00.611239 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:00.611053    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v" event={"ID":"7858edd0-1e22-469e-b1e5-45a5bb08df03","Type":"ContainerDied","Data":"82809d2b1cef96a8ed7c3e572c88229266586c7940334262d9b43d0c99804143"}
Sep 16 18:38:00.611239 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:00.611193    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82809d2b1cef96a8ed7c3e572c88229266586c7940334262d9b43d0c99804143"
Sep 16 18:38:00.955890 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:00.955852    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:01.734244 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:01.734211    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:02.426782 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:02.426745    2562 patch_prober.go:28] interesting pod/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 16 18:38:02.427193 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:02.426804    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst" podUID="39d27bcc-faf3-463c-8818-ac52063d2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 16 18:38:02.625684 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:02.625580    2562 generic.go:356] "Generic (PLEG): container finished" podID="39d27bcc-faf3-463c-8818-ac52063d2831" containerID="336ac8ec429c3ed1f0551b59621179c55dd1658878a7bd042fc23c3a7480b7e8" exitCode=0
Sep 16 18:38:02.625684 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:02.625671    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst" event={"ID":"39d27bcc-faf3-463c-8818-ac52063d2831","Type":"ContainerDied","Data":"336ac8ec429c3ed1f0551b59621179c55dd1658878a7bd042fc23c3a7480b7e8"}
Sep 16 18:38:02.626153 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:02.626135    2562 scope.go:117] "RemoveContainer" containerID="336ac8ec429c3ed1f0551b59621179c55dd1658878a7bd042fc23c3a7480b7e8"
Sep 16 18:38:02.627328 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:02.627294    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:38:02.627440 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:02.627364    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:38:02.627440 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:02.627394    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:38:02.670294 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:02.670256    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:03.402986 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:03.402951    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:03.489193 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:03.489163    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:03.569860 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:03.569814    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:03.634477 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:03.634444    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst" event={"ID":"39d27bcc-faf3-463c-8818-ac52063d2831","Type":"ContainerStarted","Data":"af0bb7ca50668e41b0ad505622289a80d6afb25c7e5e3012d7ead20969a8cc84"}
Sep 16 18:38:03.682645 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:03.682518    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:03.700380 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:03.700344    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v"]
Sep 16 18:38:03.704863 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:03.704834    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b95e3be6-z6v6v"]
Sep 16 18:38:03.791054 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:03.791020    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:03.837039 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:03.837011    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:38:03.837261 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:03.837239    2562 patch_prober.go:28] interesting pod/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 16 18:38:03.837329 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:03.837292    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst" podUID="39d27bcc-faf3-463c-8818-ac52063d2831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 16 18:38:03.842881 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:03.842845    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"]
Sep 16 18:38:03.868704 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:03.868676    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:04.284729 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:04.284689    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:04.552062 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:04.551980    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7858edd0-1e22-469e-b1e5-45a5bb08df03" path="/var/lib/kubelet/pods/7858edd0-1e22-469e-b1e5-45a5bb08df03/volumes"
Sep 16 18:38:04.632635 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:04.632602    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:04.641460 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:04.641432    2562 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst" secret="" err="secret \"e2e-derive-b95e3be6-openshift-default-dockercfg-pghsz\" not found"
Sep 16 18:38:04.765604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:04.765569    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:04.841457 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:04.841370    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:38:05.646953 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:05.646912    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst" podUID="39d27bcc-faf3-463c-8818-ac52063d2831" containerName="istio-proxy" containerID="cri-o://af0bb7ca50668e41b0ad505622289a80d6afb25c7e5e3012d7ead20969a8cc84" gracePeriod=30
Sep 16 18:38:05.647434 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:05.647047    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:38:05.647992 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:05.647970    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:38:06.217778 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:06.217720    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:06.323973 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:06.323938    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:06.554125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:06.554026    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:07.355962 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:07.355893    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:08.226579 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:08.226523    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:08.990503 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:08.990468    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:10.902067 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:10.902036    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:38:11.021095 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.021055    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-workload-socket\") pod \"39d27bcc-faf3-463c-8818-ac52063d2831\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") "
Sep 16 18:38:11.021095 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.021095    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-istio-data\") pod \"39d27bcc-faf3-463c-8818-ac52063d2831\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") "
Sep 16 18:38:11.021380 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.021150    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39d27bcc-faf3-463c-8818-ac52063d2831-istio-podinfo\") pod \"39d27bcc-faf3-463c-8818-ac52063d2831\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") "
Sep 16 18:38:11.021380 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.021179    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39d27bcc-faf3-463c-8818-ac52063d2831-istio-token\") pod \"39d27bcc-faf3-463c-8818-ac52063d2831\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") "
Sep 16 18:38:11.021380 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.021209    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-workload-certs\") pod \"39d27bcc-faf3-463c-8818-ac52063d2831\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") "
Sep 16 18:38:11.021380 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.021248    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqhnq\" (UniqueName: \"kubernetes.io/projected/39d27bcc-faf3-463c-8818-ac52063d2831-kube-api-access-wqhnq\") pod \"39d27bcc-faf3-463c-8818-ac52063d2831\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") "
Sep 16 18:38:11.021380 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.021289    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-istio-envoy\") pod \"39d27bcc-faf3-463c-8818-ac52063d2831\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") "
Sep 16 18:38:11.021694 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.021404    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39d27bcc-faf3-463c-8818-ac52063d2831" (UID: "39d27bcc-faf3-463c-8818-ac52063d2831"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:38:11.021694 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.021461    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39d27bcc-faf3-463c-8818-ac52063d2831-istiod-ca-cert\") pod \"39d27bcc-faf3-463c-8818-ac52063d2831\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") "
Sep 16 18:38:11.021694 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.021500    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-credential-socket\") pod \"39d27bcc-faf3-463c-8818-ac52063d2831\" (UID: \"39d27bcc-faf3-463c-8818-ac52063d2831\") "
Sep 16 18:38:11.021694 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.021652    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39d27bcc-faf3-463c-8818-ac52063d2831" (UID: "39d27bcc-faf3-463c-8818-ac52063d2831"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:38:11.021910 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.021833    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d27bcc-faf3-463c-8818-ac52063d2831-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39d27bcc-faf3-463c-8818-ac52063d2831" (UID: "39d27bcc-faf3-463c-8818-ac52063d2831"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:38:11.022036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.022009    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39d27bcc-faf3-463c-8818-ac52063d2831" (UID: "39d27bcc-faf3-463c-8818-ac52063d2831"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:38:11.022232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.022036    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-istio-data" (OuterVolumeSpecName: "istio-data") pod "39d27bcc-faf3-463c-8818-ac52063d2831" (UID: "39d27bcc-faf3-463c-8818-ac52063d2831"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:38:11.022232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.022042    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39d27bcc-faf3-463c-8818-ac52063d2831-istiod-ca-cert\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:38:11.022384 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.022286    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-workload-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:38:11.022384 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.022303    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-workload-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:38:11.023595 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.023562    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39d27bcc-faf3-463c-8818-ac52063d2831-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39d27bcc-faf3-463c-8818-ac52063d2831" (UID: "39d27bcc-faf3-463c-8818-ac52063d2831"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:38:11.023595 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.023565    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d27bcc-faf3-463c-8818-ac52063d2831-kube-api-access-wqhnq" (OuterVolumeSpecName: "kube-api-access-wqhnq") pod "39d27bcc-faf3-463c-8818-ac52063d2831" (UID: "39d27bcc-faf3-463c-8818-ac52063d2831"). InnerVolumeSpecName "kube-api-access-wqhnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:38:11.023757 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.023625    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39d27bcc-faf3-463c-8818-ac52063d2831" (UID: "39d27bcc-faf3-463c-8818-ac52063d2831"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:38:11.023757 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.023707    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d27bcc-faf3-463c-8818-ac52063d2831-istio-token" (OuterVolumeSpecName: "istio-token") pod "39d27bcc-faf3-463c-8818-ac52063d2831" (UID: "39d27bcc-faf3-463c-8818-ac52063d2831"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:38:11.123864 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.123830    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-credential-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:38:11.124036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.123859    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-istio-data\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:38:11.124036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.123926    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39d27bcc-faf3-463c-8818-ac52063d2831-istio-podinfo\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:38:11.124036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.123939    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39d27bcc-faf3-463c-8818-ac52063d2831-istio-token\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:38:11.124036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.123947    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqhnq\" (UniqueName: \"kubernetes.io/projected/39d27bcc-faf3-463c-8818-ac52063d2831-kube-api-access-wqhnq\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:38:11.124036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.123958    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39d27bcc-faf3-463c-8818-ac52063d2831-istio-envoy\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:38:11.694786 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.694750    2562 generic.go:356] "Generic (PLEG): container finished" podID="39d27bcc-faf3-463c-8818-ac52063d2831" containerID="af0bb7ca50668e41b0ad505622289a80d6afb25c7e5e3012d7ead20969a8cc84" exitCode=0
Sep 16 18:38:11.694996 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.694797    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"
Sep 16 18:38:11.694996 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.694836    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst" event={"ID":"39d27bcc-faf3-463c-8818-ac52063d2831","Type":"ContainerDied","Data":"af0bb7ca50668e41b0ad505622289a80d6afb25c7e5e3012d7ead20969a8cc84"}
Sep 16 18:38:11.694996 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.694907    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst" event={"ID":"39d27bcc-faf3-463c-8818-ac52063d2831","Type":"ContainerDied","Data":"de8b08679b226083ab5be29c103a3ce159a33636c99e388f4cde1d6877f9a2a3"}
Sep 16 18:38:11.694996 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.694937    2562 scope.go:117] "RemoveContainer" containerID="af0bb7ca50668e41b0ad505622289a80d6afb25c7e5e3012d7ead20969a8cc84"
Sep 16 18:38:11.713319 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.713298    2562 scope.go:117] "RemoveContainer" containerID="336ac8ec429c3ed1f0551b59621179c55dd1658878a7bd042fc23c3a7480b7e8"
Sep 16 18:38:11.722220 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.722195    2562 scope.go:117] "RemoveContainer" containerID="af0bb7ca50668e41b0ad505622289a80d6afb25c7e5e3012d7ead20969a8cc84"
Sep 16 18:38:11.722532 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:38:11.722509    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0bb7ca50668e41b0ad505622289a80d6afb25c7e5e3012d7ead20969a8cc84\": container with ID starting with af0bb7ca50668e41b0ad505622289a80d6afb25c7e5e3012d7ead20969a8cc84 not found: ID does not exist" containerID="af0bb7ca50668e41b0ad505622289a80d6afb25c7e5e3012d7ead20969a8cc84"
Sep 16 18:38:11.722643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.722560    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0bb7ca50668e41b0ad505622289a80d6afb25c7e5e3012d7ead20969a8cc84"} err="failed to get container status \"af0bb7ca50668e41b0ad505622289a80d6afb25c7e5e3012d7ead20969a8cc84\": rpc error: code = NotFound desc = could not find container \"af0bb7ca50668e41b0ad505622289a80d6afb25c7e5e3012d7ead20969a8cc84\": container with ID starting with af0bb7ca50668e41b0ad505622289a80d6afb25c7e5e3012d7ead20969a8cc84 not found: ID does not exist"
Sep 16 18:38:11.722643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.722586    2562 scope.go:117] "RemoveContainer" containerID="336ac8ec429c3ed1f0551b59621179c55dd1658878a7bd042fc23c3a7480b7e8"
Sep 16 18:38:11.722879 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:38:11.722858    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336ac8ec429c3ed1f0551b59621179c55dd1658878a7bd042fc23c3a7480b7e8\": container with ID starting with 336ac8ec429c3ed1f0551b59621179c55dd1658878a7bd042fc23c3a7480b7e8 not found: ID does not exist" containerID="336ac8ec429c3ed1f0551b59621179c55dd1658878a7bd042fc23c3a7480b7e8"
Sep 16 18:38:11.722982 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.722883    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336ac8ec429c3ed1f0551b59621179c55dd1658878a7bd042fc23c3a7480b7e8"} err="failed to get container status \"336ac8ec429c3ed1f0551b59621179c55dd1658878a7bd042fc23c3a7480b7e8\": rpc error: code = NotFound desc = could not find container \"336ac8ec429c3ed1f0551b59621179c55dd1658878a7bd042fc23c3a7480b7e8\": container with ID starting with 336ac8ec429c3ed1f0551b59621179c55dd1658878a7bd042fc23c3a7480b7e8 not found: ID does not exist"
Sep 16 18:38:11.727319 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.727297    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"]
Sep 16 18:38:11.731017 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:11.730998    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b95e3be6-openshift-default-7c8f778f88-krbst"]
Sep 16 18:38:12.011990 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:12.011955    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:38:12.434878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:12.434786    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:12.543716 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:12.543677    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:12.552205 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:12.552177    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39d27bcc-faf3-463c-8818-ac52063d2831" path="/var/lib/kubelet/pods/39d27bcc-faf3-463c-8818-ac52063d2831/volumes"
Sep 16 18:38:12.635212 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:12.635154    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:12.743777 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:12.743739    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:13.134256 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:13.134178    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:15.824138 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.824105    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"]
Sep 16 18:38:15.824535 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.824523    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d27bcc-faf3-463c-8818-ac52063d2831" containerName="istio-proxy"
Sep 16 18:38:15.824604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.824537    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d27bcc-faf3-463c-8818-ac52063d2831" containerName="istio-proxy"
Sep 16 18:38:15.824604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.824565    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7858edd0-1e22-469e-b1e5-45a5bb08df03" containerName="revoke-keys"
Sep 16 18:38:15.824604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.824571    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7858edd0-1e22-469e-b1e5-45a5bb08df03" containerName="revoke-keys"
Sep 16 18:38:15.824604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.824586    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d27bcc-faf3-463c-8818-ac52063d2831" containerName="istio-proxy"
Sep 16 18:38:15.824604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.824591    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d27bcc-faf3-463c-8818-ac52063d2831" containerName="istio-proxy"
Sep 16 18:38:15.824777 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.824657    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d27bcc-faf3-463c-8818-ac52063d2831" containerName="istio-proxy"
Sep 16 18:38:15.824777 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.824664    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7858edd0-1e22-469e-b1e5-45a5bb08df03" containerName="revoke-keys"
Sep 16 18:38:15.824777 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.824673    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d27bcc-faf3-463c-8818-ac52063d2831" containerName="istio-proxy"
Sep 16 18:38:15.827731 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.827713    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:15.830560 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.830523    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d8a600\"/\"premium-w4-d8a600-kserve-self-signed-certs\""
Sep 16 18:38:15.839384 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.839356    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"]
Sep 16 18:38:15.850866 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.850827    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:15.974741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.974698    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-dshm\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:15.974741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.974747    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ef68f3a-a4c3-4b67-9d44-2de265669783-tls-certs\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:15.974962 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.974779    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-home\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:15.974962 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.974799    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td87x\" (UniqueName: \"kubernetes.io/projected/8ef68f3a-a4c3-4b67-9d44-2de265669783-kube-api-access-td87x\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:15.974962 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.974819    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-model-cache\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:15.974962 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:15.974834    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-tmp-dir\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:16.076156 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.076066    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-dshm\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:16.076156 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.076115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ef68f3a-a4c3-4b67-9d44-2de265669783-tls-certs\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:16.076381 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.076237    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-home\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:16.076381 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.076278    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td87x\" (UniqueName: \"kubernetes.io/projected/8ef68f3a-a4c3-4b67-9d44-2de265669783-kube-api-access-td87x\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:16.076504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.076405    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-model-cache\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:16.076504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.076445    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-tmp-dir\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:16.076681 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.076659    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-home\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:16.076786 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.076761    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-model-cache\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:16.076834 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.076790    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-tmp-dir\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:16.078526 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.078502    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-dshm\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:16.078756 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.078740    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ef68f3a-a4c3-4b67-9d44-2de265669783-tls-certs\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:16.087956 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.087925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td87x\" (UniqueName: \"kubernetes.io/projected/8ef68f3a-a4c3-4b67-9d44-2de265669783-kube-api-access-td87x\") pod \"premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") " pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:16.137566 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.137512    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:16.423118 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.423089    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"]
Sep 16 18:38:16.423974 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:38:16.423947    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ef68f3a_a4c3_4b67_9d44_2de265669783.slice/crio-22d1b4e97ce01916d3b26bd907640b3d5ff703d5ef66e161bd77abecc63e4d14 WatchSource:0}: Error finding container 22d1b4e97ce01916d3b26bd907640b3d5ff703d5ef66e161bd77abecc63e4d14: Status 404 returned error can't find the container with id 22d1b4e97ce01916d3b26bd907640b3d5ff703d5ef66e161bd77abecc63e4d14
Sep 16 18:38:16.729721 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.729688    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5" event={"ID":"8ef68f3a-a4c3-4b67-9d44-2de265669783","Type":"ContainerStarted","Data":"de1d5422e29e83823f8081fc51ed6f1892260914d2e48d3a9e41fc350a76433d"}
Sep 16 18:38:16.729883 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.729731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5" event={"ID":"8ef68f3a-a4c3-4b67-9d44-2de265669783","Type":"ContainerStarted","Data":"22d1b4e97ce01916d3b26bd907640b3d5ff703d5ef66e161bd77abecc63e4d14"}
Sep 16 18:38:16.754813 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:16.754747    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5" podStartSLOduration=1.754725911 podStartE2EDuration="1.754725911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:38:16.750076591 +0000 UTC m=+1334.614980640" watchObservedRunningTime="2026-09-16 18:38:16.754725911 +0000 UTC m=+1334.619629960"
Sep 16 18:38:26.138178 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:26.138078    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:26.138178 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:26.138151    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:26.151828 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:26.151790    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:26.808041 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:26.808008    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:38:26.924193 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:26.924151    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"]
Sep 16 18:38:26.927438 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:26.927413    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:26.930257 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:26.930227    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6ef84e\"/\"openshift-service-ca.crt\""
Sep 16 18:38:26.930437 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:26.930263    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6ef84e\"/\"test-model-6ef84e-kserve-self-signed-certs\""
Sep 16 18:38:26.930437 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:26.930272    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6ef84e\"/\"default-dockercfg-cxfll\""
Sep 16 18:38:26.930628 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:26.930329    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6ef84e\"/\"kube-root-ca.crt\""
Sep 16 18:38:26.935817 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:26.935792    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"]
Sep 16 18:38:26.966419 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:26.966381    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:27.089769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.089671    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793d66ce-acdd-4042-8cea-1b4f64f01e37-tls-certs\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.089769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.089718    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-model-cache\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.089769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.089746    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd58r\" (UniqueName: \"kubernetes.io/projected/793d66ce-acdd-4042-8cea-1b4f64f01e37-kube-api-access-xd58r\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.090043 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.089832    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-dshm\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.090043 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.089872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-home\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.090043 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.089957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-tmp-dir\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.191324 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.191288    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-dshm\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.191814 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.191335    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-home\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.191814 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.191374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-tmp-dir\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.191814 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.191405    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793d66ce-acdd-4042-8cea-1b4f64f01e37-tls-certs\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.191814 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.191425    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-model-cache\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.191814 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.191446    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd58r\" (UniqueName: \"kubernetes.io/projected/793d66ce-acdd-4042-8cea-1b4f64f01e37-kube-api-access-xd58r\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.192050 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.191845    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-home\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.192050 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.191903    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-tmp-dir\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.192050 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.191925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-model-cache\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.193731 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.193703    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-dshm\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.194058 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.194037    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793d66ce-acdd-4042-8cea-1b4f64f01e37-tls-certs\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.202094 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.202070    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd58r\" (UniqueName: \"kubernetes.io/projected/793d66ce-acdd-4042-8cea-1b4f64f01e37-kube-api-access-xd58r\") pod \"test-model-6ef84e-kserve-dd85b7d9-cnhtr\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") " pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.241001 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.240968    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:27.516189 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.516159    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"]
Sep 16 18:38:27.517519 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:38:27.517488    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod793d66ce_acdd_4042_8cea_1b4f64f01e37.slice/crio-d2246d01485fc501f5f187426afd786686b80e20fb44c159acc0b45e0c3a0443 WatchSource:0}: Error finding container d2246d01485fc501f5f187426afd786686b80e20fb44c159acc0b45e0c3a0443: Status 404 returned error can't find the container with id d2246d01485fc501f5f187426afd786686b80e20fb44c159acc0b45e0c3a0443
Sep 16 18:38:27.804770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.804678    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr" event={"ID":"793d66ce-acdd-4042-8cea-1b4f64f01e37","Type":"ContainerStarted","Data":"a2635781b4f91f8a73a34046aed853e1da09240674f1928a7cc73022833767ad"}
Sep 16 18:38:27.804770 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.804722    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr" event={"ID":"793d66ce-acdd-4042-8cea-1b4f64f01e37","Type":"ContainerStarted","Data":"d2246d01485fc501f5f187426afd786686b80e20fb44c159acc0b45e0c3a0443"}
Sep 16 18:38:27.828707 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:27.828654    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr" podStartSLOduration=1.828636924 podStartE2EDuration="1.828636924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:38:27.825887507 +0000 UTC m=+1345.690791556" watchObservedRunningTime="2026-09-16 18:38:27.828636924 +0000 UTC m=+1345.693540993"
Sep 16 18:38:35.415865 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:35.415824    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:38:37.241600 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:37.241537    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:37.242037 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:37.241614    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:37.258430 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:37.258393    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:37.887606 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:37.887560    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:38:39.104387 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:39.104355    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:39.988065 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:39.988030    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:45.524236 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:45.524197    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:45.632019 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:45.631973    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:45.782840 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:45.782754    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:46.999242 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:46.999206    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:57.519427 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:38:57.519386    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:39:01.685264 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:01.685226    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:08.036199 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:08.036145    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:08.184721 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:08.184662    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:08.281534 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:08.281498    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:13.285701 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:13.285664    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:13.481320 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:13.481283    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2"]
Sep 16 18:39:13.486047 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:13.486026    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2"
Sep 16 18:39:13.489479 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:13.489449    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:39:13.490051 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:13.490032    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:39:13.500819 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:13.500781    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2"]
Sep 16 18:39:13.576897 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:13.576804    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2\" (UID: \"e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2"
Sep 16 18:39:13.643260 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:13.643227    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:13.677949 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:13.677903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2\" (UID: \"e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2"
Sep 16 18:39:13.678700 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:13.678677    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2\" (UID: \"e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2"
Sep 16 18:39:13.796680 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:13.796636    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2"
Sep 16 18:39:14.079493 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:14.079468    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2"]
Sep 16 18:39:14.082009 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:39:14.081971    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53e6d0c_aac7_4bf0_a7f6_1b795c7daa8b.slice/crio-6c0fa89569d6dfc24e76c96cefef39f38e9e2e173104ed52019ccf168063ff94 WatchSource:0}: Error finding container 6c0fa89569d6dfc24e76c96cefef39f38e9e2e173104ed52019ccf168063ff94: Status 404 returned error can't find the container with id 6c0fa89569d6dfc24e76c96cefef39f38e9e2e173104ed52019ccf168063ff94
Sep 16 18:39:14.127600 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:14.127560    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2" event={"ID":"e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b","Type":"ContainerStarted","Data":"6c0fa89569d6dfc24e76c96cefef39f38e9e2e173104ed52019ccf168063ff94"}
Sep 16 18:39:14.573628 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:14.573572    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:15.136030 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:15.135932    2562 generic.go:356] "Generic (PLEG): container finished" podID="e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b" containerID="12bbfe10cc69b29f37ab0483cd447f6d697edf1f7f58d748f60a1ec0cf07e477" exitCode=0
Sep 16 18:39:15.136197 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:15.136023    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2" event={"ID":"e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b","Type":"ContainerDied","Data":"12bbfe10cc69b29f37ab0483cd447f6d697edf1f7f58d748f60a1ec0cf07e477"}
Sep 16 18:39:15.487400 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:15.487369    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:16.288986 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:16.288963    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2"
Sep 16 18:39:16.407812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:16.407724    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b-maas-api-service-ca\") pod \"e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b\" (UID: \"e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b\") "
Sep 16 18:39:16.408123 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:16.408100    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b" (UID: "e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:39:16.509312 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:16.509264    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b-maas-api-service-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:39:16.653166 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:16.653128    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:17.151007 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:17.150975    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2" event={"ID":"e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b","Type":"ContainerDied","Data":"6c0fa89569d6dfc24e76c96cefef39f38e9e2e173104ed52019ccf168063ff94"}
Sep 16 18:39:17.151007 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:17.150996    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2"
Sep 16 18:39:17.151282 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:17.151029    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c0fa89569d6dfc24e76c96cefef39f38e9e2e173104ed52019ccf168063ff94"
Sep 16 18:39:17.735277 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:17.735243    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:18.765837 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:18.765801    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:18.920679 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:18.920645    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:19.552702 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:19.552662    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:19.655328 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:19.655294    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:24.104444 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:24.104408    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:24.203424 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:24.203393    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:24.303920 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:24.303889    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:24.350275 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:24.350239    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2"]
Sep 16 18:39:24.353857 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:24.353823    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efa70c-gk9m2"]
Sep 16 18:39:24.476787 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:24.476748    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:24.553148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:24.553111    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b" path="/var/lib/kubelet/pods/e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b/volumes"
Sep 16 18:39:24.728398 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:24.728312    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:24.864776 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:24.864743    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:25.841364 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:25.841325    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:27.163327 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:27.163287    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:28.022822 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:28.022780    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:28.189675 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:28.189630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:28.292862 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:28.292764    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:28.404070 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:28.404037    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:28.515361 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:28.515325    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:29.200873 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:29.200841    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:29.321303 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:29.321265    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:29.439407 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:29.439368    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:29.610896 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:29.610799    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:30.945280 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:30.945242    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:31.736129 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:31.736092    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:39:32.307893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:32.307858    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:32.380373 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:32.380340    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:32.608511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:32.608422    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:33.051750 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:33.051714    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:33.441253 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:33.441168    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:33.934722 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:33.934670    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:34.935727 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:34.935696    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:35.033472 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:35.033431    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:35.138947 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:35.138914    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:38.448651 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:38.448606    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:38.572821 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:38.572771    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:39.205323 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:39.205285    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:45.766405 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:45.766369    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:45.862934 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:45.862899    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:46.040810 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:46.040725    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:46.475241 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:46.473957    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:46.637339 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:46.637304    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:46.784590 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:46.784494    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:49.054539 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:49.054496    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:49.727360 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:49.727322    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:49.812986 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:49.812951    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:49.996726 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:49.996631    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:50.768040 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:50.768006    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:51.219313 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:51.219280    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:52.091344 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:52.091312    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:53.231678 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:53.231644    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:54.205538 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:54.205501    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:54.300382 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:54.300348    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:54.837567 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:54.837481    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:54.941229 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:54.941194    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:55.037372 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:55.037339    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:56.121552 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:39:56.121506    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:00.644530 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:00.644490    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:03.061098 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:03.061059    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:05.623645 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:05.623613    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:11.275084 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:11.275047    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:11.350724 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:11.350675    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:40:15.561486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:15.561448    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:17.168490 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:17.168459    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:18.142952 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:18.142902    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:19.218984 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:19.218951    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:19.805104 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:19.805074    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:20.119435 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:20.119345    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:20.556156 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:20.556116    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:20.723295 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:20.723263    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:21.048482 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:21.048452    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:21.615821 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:21.615786    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:22.652920 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:22.652884    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:29.353666 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:29.353587    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:29.439829 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:29.439788    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:29.768214 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:29.768176    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j"]
Sep 16 18:40:29.768661 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:29.768646    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b" containerName="revoke-keys"
Sep 16 18:40:29.768718 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:29.768662    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b" containerName="revoke-keys"
Sep 16 18:40:29.768755 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:29.768726    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e53e6d0c-aac7-4bf0-a7f6-1b795c7daa8b" containerName="revoke-keys"
Sep 16 18:40:29.772088 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:29.772071    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j"
Sep 16 18:40:29.774644 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:29.774620    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:40:29.775113 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:29.775094    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:40:29.785627 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:29.785598    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j"]
Sep 16 18:40:29.861176 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:29.861142    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4281da-c21b-4fba-84f8-09a680c11ae3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j\" (UID: \"ab4281da-c21b-4fba-84f8-09a680c11ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j"
Sep 16 18:40:29.962148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:29.962106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4281da-c21b-4fba-84f8-09a680c11ae3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j\" (UID: \"ab4281da-c21b-4fba-84f8-09a680c11ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j"
Sep 16 18:40:29.962947 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:29.962924    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4281da-c21b-4fba-84f8-09a680c11ae3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j\" (UID: \"ab4281da-c21b-4fba-84f8-09a680c11ae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j"
Sep 16 18:40:30.084361 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:30.084264    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j"
Sep 16 18:40:30.399081 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:30.399047    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j"]
Sep 16 18:40:30.400306 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:40:30.400280    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4281da_c21b_4fba_84f8_09a680c11ae3.slice/crio-7912464b18122a9b8bfde630106e105ffc7c58ab2b2d30d8634c41ed00f1ed17 WatchSource:0}: Error finding container 7912464b18122a9b8bfde630106e105ffc7c58ab2b2d30d8634c41ed00f1ed17: Status 404 returned error can't find the container with id 7912464b18122a9b8bfde630106e105ffc7c58ab2b2d30d8634c41ed00f1ed17
Sep 16 18:40:30.647433 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:30.647340    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j" event={"ID":"ab4281da-c21b-4fba-84f8-09a680c11ae3","Type":"ContainerStarted","Data":"7912464b18122a9b8bfde630106e105ffc7c58ab2b2d30d8634c41ed00f1ed17"}
Sep 16 18:40:30.858626 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:30.858588    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:31.656833 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:31.656794    2562 generic.go:356] "Generic (PLEG): container finished" podID="ab4281da-c21b-4fba-84f8-09a680c11ae3" containerID="24f9d253e1fc34561f1bce315b290ae85a03838795a9063e982367e07d4732e8" exitCode=0
Sep 16 18:40:31.657234 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:31.656877    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j" event={"ID":"ab4281da-c21b-4fba-84f8-09a680c11ae3","Type":"ContainerDied","Data":"24f9d253e1fc34561f1bce315b290ae85a03838795a9063e982367e07d4732e8"}
Sep 16 18:40:31.873972 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:31.873936    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:32.711760 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:32.711730    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:32.825233 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:32.825209    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j"
Sep 16 18:40:32.891384 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:32.891342    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4281da-c21b-4fba-84f8-09a680c11ae3-maas-api-service-ca\") pod \"ab4281da-c21b-4fba-84f8-09a680c11ae3\" (UID: \"ab4281da-c21b-4fba-84f8-09a680c11ae3\") "
Sep 16 18:40:32.891757 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:32.891730    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4281da-c21b-4fba-84f8-09a680c11ae3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ab4281da-c21b-4fba-84f8-09a680c11ae3" (UID: "ab4281da-c21b-4fba-84f8-09a680c11ae3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:40:32.891839 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:32.891825    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab4281da-c21b-4fba-84f8-09a680c11ae3-maas-api-service-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:40:33.576492 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:33.576454    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:33.670812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:33.670776    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j"
Sep 16 18:40:33.670812 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:33.670803    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j" event={"ID":"ab4281da-c21b-4fba-84f8-09a680c11ae3","Type":"ContainerDied","Data":"7912464b18122a9b8bfde630106e105ffc7c58ab2b2d30d8634c41ed00f1ed17"}
Sep 16 18:40:33.671030 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:33.670851    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7912464b18122a9b8bfde630106e105ffc7c58ab2b2d30d8634c41ed00f1ed17"
Sep 16 18:40:34.504799 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:34.504764    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:35.276044 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:35.276000    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:38.853111 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.853062    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"]
Sep 16 18:40:38.853728 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.853706    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4281da-c21b-4fba-84f8-09a680c11ae3" containerName="revoke-keys"
Sep 16 18:40:38.853821 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.853730    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4281da-c21b-4fba-84f8-09a680c11ae3" containerName="revoke-keys"
Sep 16 18:40:38.853886 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.853861    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab4281da-c21b-4fba-84f8-09a680c11ae3" containerName="revoke-keys"
Sep 16 18:40:38.857164 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.857139    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:38.861847 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.861800    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c8b23c\"/\"openshift-service-ca.crt\""
Sep 16 18:40:38.861847 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.861814    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c8b23c\"/\"kube-root-ca.crt\""
Sep 16 18:40:38.862089 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.861840    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c8b23c\"/\"noten-model-c8b23c-kserve-self-signed-certs\""
Sep 16 18:40:38.862089 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.861952    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c8b23c\"/\"default-dockercfg-vjn6j\""
Sep 16 18:40:38.881761 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.881730    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"]
Sep 16 18:40:38.946058 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.946024    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:38.953963 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.953922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-tmp-dir\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:38.954147 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.953971    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5cld\" (UniqueName: \"kubernetes.io/projected/9e429196-411e-4152-9084-d9f5d9af7a40-kube-api-access-p5cld\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:38.954147 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.954008    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-dshm\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:38.954147 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.954055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e429196-411e-4152-9084-d9f5d9af7a40-tls-certs\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:38.954147 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.954133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-model-cache\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:38.954390 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:38.954157    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-home\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:39.055057 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.055018    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-tmp-dir\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:39.055057 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.055068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5cld\" (UniqueName: \"kubernetes.io/projected/9e429196-411e-4152-9084-d9f5d9af7a40-kube-api-access-p5cld\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:39.055341 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.055105    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-dshm\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:39.055341 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.055179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e429196-411e-4152-9084-d9f5d9af7a40-tls-certs\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:39.055341 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.055252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-model-cache\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:39.055341 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.055280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-home\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:39.055666 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.055632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-tmp-dir\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:39.055824 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.055748    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-model-cache\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:39.055824 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.055654    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-home\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:39.057889 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.057861    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-dshm\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:39.058315 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.058296    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e429196-411e-4152-9084-d9f5d9af7a40-tls-certs\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:39.073599 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.073567    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5cld\" (UniqueName: \"kubernetes.io/projected/9e429196-411e-4152-9084-d9f5d9af7a40-kube-api-access-p5cld\") pod \"noten-model-c8b23c-kserve-68d9db66ff-tpz5g\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") " pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:39.167305 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.167201    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:39.461928 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.461890    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"]
Sep 16 18:40:39.467196 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:40:39.467164    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e429196_411e_4152_9084_d9f5d9af7a40.slice/crio-8fe9f04c2ef8348efdab697cf31f0c9d0d2b8715c6e658f749d42a0f6ec5529f WatchSource:0}: Error finding container 8fe9f04c2ef8348efdab697cf31f0c9d0d2b8715c6e658f749d42a0f6ec5529f: Status 404 returned error can't find the container with id 8fe9f04c2ef8348efdab697cf31f0c9d0d2b8715c6e658f749d42a0f6ec5529f
Sep 16 18:40:39.709862 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.709825    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g" event={"ID":"9e429196-411e-4152-9084-d9f5d9af7a40","Type":"ContainerStarted","Data":"5c2da46da0d7f49f539ad030d8eb2ca3cad74bedf13264d2fbdb5861446ee706"}
Sep 16 18:40:39.710047 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.709875    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g" event={"ID":"9e429196-411e-4152-9084-d9f5d9af7a40","Type":"ContainerStarted","Data":"8fe9f04c2ef8348efdab697cf31f0c9d0d2b8715c6e658f749d42a0f6ec5529f"}
Sep 16 18:40:39.756244 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:39.756112    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g" podStartSLOduration=1.756080689 podStartE2EDuration="1.756080689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:39.755227381 +0000 UTC m=+1477.620131429" watchObservedRunningTime="2026-09-16 18:40:39.756080689 +0000 UTC m=+1477.620984741"
Sep 16 18:40:40.651418 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:40.651381    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:40.835252 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:40.835215    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:40.965155 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:40.965113    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:41.256261 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:41.255984    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j"]
Sep 16 18:40:41.259487 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:41.259450    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-58beb43a-s8n6j"]
Sep 16 18:40:41.259679 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:41.259496    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:41.443462 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:41.443429    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:41.633593 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:41.633473    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:41.766167 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:41.766129    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:41.802075 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:41.802031    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:40:42.554357 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:42.554312    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab4281da-c21b-4fba-84f8-09a680c11ae3" path="/var/lib/kubelet/pods/ab4281da-c21b-4fba-84f8-09a680c11ae3/volumes"
Sep 16 18:40:43.177819 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:43.177767    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:44.356356 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:44.356319    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:47.162392 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:47.162328    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:47.780122 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:47.780080    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:49.167629 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:49.167589    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:49.168003 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:49.167652    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:49.180513 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:49.180480    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:49.788502 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:49.788468    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:40:50.727734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:50.727695    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:50.845297 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:50.845259    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:51.141758 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:51.141669    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:51.507708 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:51.507659    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:51.665751 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:51.665711    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:51.838444 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:51.838354    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:56.929860 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:56.929819    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"]
Sep 16 18:40:56.950520 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:56.950446    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:59.231688 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:59.231655    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:59.345713 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:59.345675    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:59.433494 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:59.433463    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:59.583620 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:40:59.583482    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:02.657897 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:02.657862    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:41:02.658437 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:02.658102    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:41:02.665126 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:02.665100    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:41:02.665503 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:02.665487    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:41:03.781643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:03.781591    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:04.834517 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:04.834480    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"]
Sep 16 18:41:04.853468 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:04.853433    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:05.248633 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:05.248595    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:06.112872 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:06.112829    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"]
Sep 16 18:41:06.116233 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:06.116195    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:06.263567 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:06.263507    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:06.468209 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:06.468173    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:06.675966 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:06.675922    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:07.104868 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:07.104835    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:09.828894 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:09.828860    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:11.626674 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:11.626635    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:11.810527 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:11.810498    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:11.949926 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:11.949882    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd" podUID="05c7d397-a627-4768-b2ef-42824aceae16" containerName="main" containerID="cri-o://edf730514df422d015576b76ee162043d52d34fddb894241d7e0a23c54447977" gracePeriod=45
Sep 16 18:41:12.198645 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.198619    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:41:12.311996 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.311897    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfd6m\" (UniqueName: \"kubernetes.io/projected/05c7d397-a627-4768-b2ef-42824aceae16-kube-api-access-bfd6m\") pod \"05c7d397-a627-4768-b2ef-42824aceae16\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") "
Sep 16 18:41:12.312179 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.311995    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-home\") pod \"05c7d397-a627-4768-b2ef-42824aceae16\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") "
Sep 16 18:41:12.312179 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.312047    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05c7d397-a627-4768-b2ef-42824aceae16-tls-certs\") pod \"05c7d397-a627-4768-b2ef-42824aceae16\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") "
Sep 16 18:41:12.312179 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.312107    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-model-cache\") pod \"05c7d397-a627-4768-b2ef-42824aceae16\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") "
Sep 16 18:41:12.312179 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.312130    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-tmp-dir\") pod \"05c7d397-a627-4768-b2ef-42824aceae16\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") "
Sep 16 18:41:12.312179 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.312152    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-dshm\") pod \"05c7d397-a627-4768-b2ef-42824aceae16\" (UID: \"05c7d397-a627-4768-b2ef-42824aceae16\") "
Sep 16 18:41:12.312434 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.312304    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-home" (OuterVolumeSpecName: "home") pod "05c7d397-a627-4768-b2ef-42824aceae16" (UID: "05c7d397-a627-4768-b2ef-42824aceae16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:12.312434 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.312323    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-model-cache" (OuterVolumeSpecName: "model-cache") pod "05c7d397-a627-4768-b2ef-42824aceae16" (UID: "05c7d397-a627-4768-b2ef-42824aceae16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:12.312579 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.312425    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05c7d397-a627-4768-b2ef-42824aceae16" (UID: "05c7d397-a627-4768-b2ef-42824aceae16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:12.312644 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.312610    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:12.312644 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.312627    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:12.312644 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.312637    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:12.314357 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.314327    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05c7d397-a627-4768-b2ef-42824aceae16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05c7d397-a627-4768-b2ef-42824aceae16" (UID: "05c7d397-a627-4768-b2ef-42824aceae16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:41:12.314357 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.314348    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c7d397-a627-4768-b2ef-42824aceae16-kube-api-access-bfd6m" (OuterVolumeSpecName: "kube-api-access-bfd6m") pod "05c7d397-a627-4768-b2ef-42824aceae16" (UID: "05c7d397-a627-4768-b2ef-42824aceae16"). InnerVolumeSpecName "kube-api-access-bfd6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:12.314511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.314330    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-dshm" (OuterVolumeSpecName: "dshm") pod "05c7d397-a627-4768-b2ef-42824aceae16" (UID: "05c7d397-a627-4768-b2ef-42824aceae16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:12.413239 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.413199    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfd6m\" (UniqueName: \"kubernetes.io/projected/05c7d397-a627-4768-b2ef-42824aceae16-kube-api-access-bfd6m\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:12.413239 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.413232    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05c7d397-a627-4768-b2ef-42824aceae16-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:12.413239 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.413243    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05c7d397-a627-4768-b2ef-42824aceae16-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:12.804275 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.804244    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:12.937725 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.937691    2562 generic.go:356] "Generic (PLEG): container finished" podID="05c7d397-a627-4768-b2ef-42824aceae16" containerID="edf730514df422d015576b76ee162043d52d34fddb894241d7e0a23c54447977" exitCode=0
Sep 16 18:41:12.937895 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.937736    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"
Sep 16 18:41:12.937895 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.937738    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd" event={"ID":"05c7d397-a627-4768-b2ef-42824aceae16","Type":"ContainerDied","Data":"edf730514df422d015576b76ee162043d52d34fddb894241d7e0a23c54447977"}
Sep 16 18:41:12.937895 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.937846    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd" event={"ID":"05c7d397-a627-4768-b2ef-42824aceae16","Type":"ContainerDied","Data":"4345701a35beb0b1b578d98c62b932a713adf326ce6ca7082155818879b19b11"}
Sep 16 18:41:12.937895 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.937868    2562 scope.go:117] "RemoveContainer" containerID="edf730514df422d015576b76ee162043d52d34fddb894241d7e0a23c54447977"
Sep 16 18:41:12.947708 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.947682    2562 scope.go:117] "RemoveContainer" containerID="edf730514df422d015576b76ee162043d52d34fddb894241d7e0a23c54447977"
Sep 16 18:41:12.948032 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:41:12.948011    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf730514df422d015576b76ee162043d52d34fddb894241d7e0a23c54447977\": container with ID starting with edf730514df422d015576b76ee162043d52d34fddb894241d7e0a23c54447977 not found: ID does not exist" containerID="edf730514df422d015576b76ee162043d52d34fddb894241d7e0a23c54447977"
Sep 16 18:41:12.948098 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.948040    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf730514df422d015576b76ee162043d52d34fddb894241d7e0a23c54447977"} err="failed to get container status \"edf730514df422d015576b76ee162043d52d34fddb894241d7e0a23c54447977\": rpc error: code = NotFound desc = could not find container \"edf730514df422d015576b76ee162043d52d34fddb894241d7e0a23c54447977\": container with ID starting with edf730514df422d015576b76ee162043d52d34fddb894241d7e0a23c54447977 not found: ID does not exist"
Sep 16 18:41:12.975847 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.975799    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:12.994485 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:12.994446    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"]
Sep 16 18:41:13.006441 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:13.006409    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4ab744/test-model-4ab744-kserve-6d7689585-5rsvd"]
Sep 16 18:41:13.121498 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:13.121411    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:13.877980 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:13.877946    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:14.409884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:14.409846    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:14.551621 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:14.551582    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05c7d397-a627-4768-b2ef-42824aceae16" path="/var/lib/kubelet/pods/05c7d397-a627-4768-b2ef-42824aceae16/volumes"
Sep 16 18:41:14.559009 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:14.558974    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:14.682120 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:14.682038    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:15.251779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:15.251736    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:41:16.553791 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:16.553754    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:16.903114 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:16.903034    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:17.107461 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:17.107430    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:17.444643 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:17.444601    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:17.677565 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:17.677494    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:17.869755 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:17.869723    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:18.170275 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:18.170188    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:19.590157 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:19.590125    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:19.855123 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:19.855030    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr" podUID="793d66ce-acdd-4042-8cea-1b4f64f01e37" containerName="main" containerID="cri-o://a2635781b4f91f8a73a34046aed853e1da09240674f1928a7cc73022833767ad" gracePeriod=45
Sep 16 18:41:19.995064 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:19.995030    2562 generic.go:356] "Generic (PLEG): container finished" podID="793d66ce-acdd-4042-8cea-1b4f64f01e37" containerID="a2635781b4f91f8a73a34046aed853e1da09240674f1928a7cc73022833767ad" exitCode=0
Sep 16 18:41:19.995222 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:19.995117    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr" event={"ID":"793d66ce-acdd-4042-8cea-1b4f64f01e37","Type":"ContainerDied","Data":"a2635781b4f91f8a73a34046aed853e1da09240674f1928a7cc73022833767ad"}
Sep 16 18:41:20.110890 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.110835    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:41:20.195660 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.195620    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-tmp-dir\") pod \"793d66ce-acdd-4042-8cea-1b4f64f01e37\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") "
Sep 16 18:41:20.195838 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.195696    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd58r\" (UniqueName: \"kubernetes.io/projected/793d66ce-acdd-4042-8cea-1b4f64f01e37-kube-api-access-xd58r\") pod \"793d66ce-acdd-4042-8cea-1b4f64f01e37\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") "
Sep 16 18:41:20.195838 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.195767    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-model-cache\") pod \"793d66ce-acdd-4042-8cea-1b4f64f01e37\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") "
Sep 16 18:41:20.195838 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.195798    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-home\") pod \"793d66ce-acdd-4042-8cea-1b4f64f01e37\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") "
Sep 16 18:41:20.195838 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.195837    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-dshm\") pod \"793d66ce-acdd-4042-8cea-1b4f64f01e37\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") "
Sep 16 18:41:20.196069 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.195867    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793d66ce-acdd-4042-8cea-1b4f64f01e37-tls-certs\") pod \"793d66ce-acdd-4042-8cea-1b4f64f01e37\" (UID: \"793d66ce-acdd-4042-8cea-1b4f64f01e37\") "
Sep 16 18:41:20.196069 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.195916    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "793d66ce-acdd-4042-8cea-1b4f64f01e37" (UID: "793d66ce-acdd-4042-8cea-1b4f64f01e37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:20.196167 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.196082    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-home" (OuterVolumeSpecName: "home") pod "793d66ce-acdd-4042-8cea-1b4f64f01e37" (UID: "793d66ce-acdd-4042-8cea-1b4f64f01e37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:20.196167 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.196097    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-model-cache" (OuterVolumeSpecName: "model-cache") pod "793d66ce-acdd-4042-8cea-1b4f64f01e37" (UID: "793d66ce-acdd-4042-8cea-1b4f64f01e37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:20.196276 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.196214    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:20.196276 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.196228    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:20.196276 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.196238    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:20.198028 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.197997    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-dshm" (OuterVolumeSpecName: "dshm") pod "793d66ce-acdd-4042-8cea-1b4f64f01e37" (UID: "793d66ce-acdd-4042-8cea-1b4f64f01e37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:20.198150 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.198051    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793d66ce-acdd-4042-8cea-1b4f64f01e37-kube-api-access-xd58r" (OuterVolumeSpecName: "kube-api-access-xd58r") pod "793d66ce-acdd-4042-8cea-1b4f64f01e37" (UID: "793d66ce-acdd-4042-8cea-1b4f64f01e37"). InnerVolumeSpecName "kube-api-access-xd58r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:20.198150 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.198057    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/793d66ce-acdd-4042-8cea-1b4f64f01e37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "793d66ce-acdd-4042-8cea-1b4f64f01e37" (UID: "793d66ce-acdd-4042-8cea-1b4f64f01e37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:41:20.297076 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.297044    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd58r\" (UniqueName: \"kubernetes.io/projected/793d66ce-acdd-4042-8cea-1b4f64f01e37-kube-api-access-xd58r\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:20.297076 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.297074    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793d66ce-acdd-4042-8cea-1b4f64f01e37-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:20.297295 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.297087    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793d66ce-acdd-4042-8cea-1b4f64f01e37-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:20.938660 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:20.938558    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:21.003405 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.003373    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr" event={"ID":"793d66ce-acdd-4042-8cea-1b4f64f01e37","Type":"ContainerDied","Data":"d2246d01485fc501f5f187426afd786686b80e20fb44c159acc0b45e0c3a0443"}
Sep 16 18:41:21.003602 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.003419    2562 scope.go:117] "RemoveContainer" containerID="a2635781b4f91f8a73a34046aed853e1da09240674f1928a7cc73022833767ad"
Sep 16 18:41:21.003602 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.003379    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"
Sep 16 18:41:21.039469 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.039436    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"]
Sep 16 18:41:21.058020 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.057989    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6ef84e/test-model-6ef84e-kserve-dd85b7d9-cnhtr"]
Sep 16 18:41:21.132908 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.132868    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g" podUID="9e429196-411e-4152-9084-d9f5d9af7a40" containerName="main" containerID="cri-o://5c2da46da0d7f49f539ad030d8eb2ca3cad74bedf13264d2fbdb5861446ee706" gracePeriod=45
Sep 16 18:41:21.389117 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.389094    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:41:21.509232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.509155    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e429196-411e-4152-9084-d9f5d9af7a40-tls-certs\") pod \"9e429196-411e-4152-9084-d9f5d9af7a40\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") "
Sep 16 18:41:21.509232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.509218    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-model-cache\") pod \"9e429196-411e-4152-9084-d9f5d9af7a40\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") "
Sep 16 18:41:21.509443 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.509271    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-tmp-dir\") pod \"9e429196-411e-4152-9084-d9f5d9af7a40\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") "
Sep 16 18:41:21.509443 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.509396    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5cld\" (UniqueName: \"kubernetes.io/projected/9e429196-411e-4152-9084-d9f5d9af7a40-kube-api-access-p5cld\") pod \"9e429196-411e-4152-9084-d9f5d9af7a40\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") "
Sep 16 18:41:21.509525 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.509442    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-home\") pod \"9e429196-411e-4152-9084-d9f5d9af7a40\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") "
Sep 16 18:41:21.509525 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.509482    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-dshm\") pod \"9e429196-411e-4152-9084-d9f5d9af7a40\" (UID: \"9e429196-411e-4152-9084-d9f5d9af7a40\") "
Sep 16 18:41:21.509694 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.509521    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e429196-411e-4152-9084-d9f5d9af7a40" (UID: "9e429196-411e-4152-9084-d9f5d9af7a40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:21.509694 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.509517    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e429196-411e-4152-9084-d9f5d9af7a40" (UID: "9e429196-411e-4152-9084-d9f5d9af7a40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:21.509806 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.509755    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-home" (OuterVolumeSpecName: "home") pod "9e429196-411e-4152-9084-d9f5d9af7a40" (UID: "9e429196-411e-4152-9084-d9f5d9af7a40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:21.509922 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.509905    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:21.509966 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.509924    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:21.509966 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.509933    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:21.511646 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.511623    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e429196-411e-4152-9084-d9f5d9af7a40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e429196-411e-4152-9084-d9f5d9af7a40" (UID: "9e429196-411e-4152-9084-d9f5d9af7a40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:41:21.511646 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.511633    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-dshm" (OuterVolumeSpecName: "dshm") pod "9e429196-411e-4152-9084-d9f5d9af7a40" (UID: "9e429196-411e-4152-9084-d9f5d9af7a40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:21.511837 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.511659    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e429196-411e-4152-9084-d9f5d9af7a40-kube-api-access-p5cld" (OuterVolumeSpecName: "kube-api-access-p5cld") pod "9e429196-411e-4152-9084-d9f5d9af7a40" (UID: "9e429196-411e-4152-9084-d9f5d9af7a40"). InnerVolumeSpecName "kube-api-access-p5cld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:21.610939 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.610905    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5cld\" (UniqueName: \"kubernetes.io/projected/9e429196-411e-4152-9084-d9f5d9af7a40-kube-api-access-p5cld\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:21.610939 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.610933    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e429196-411e-4152-9084-d9f5d9af7a40-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:21.610939 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:21.610943    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e429196-411e-4152-9084-d9f5d9af7a40-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:22.011940 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:22.011900    2562 generic.go:356] "Generic (PLEG): container finished" podID="9e429196-411e-4152-9084-d9f5d9af7a40" containerID="5c2da46da0d7f49f539ad030d8eb2ca3cad74bedf13264d2fbdb5861446ee706" exitCode=0
Sep 16 18:41:22.012375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:22.011942    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"
Sep 16 18:41:22.012375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:22.011984    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g" event={"ID":"9e429196-411e-4152-9084-d9f5d9af7a40","Type":"ContainerDied","Data":"5c2da46da0d7f49f539ad030d8eb2ca3cad74bedf13264d2fbdb5861446ee706"}
Sep 16 18:41:22.012375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:22.012030    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g" event={"ID":"9e429196-411e-4152-9084-d9f5d9af7a40","Type":"ContainerDied","Data":"8fe9f04c2ef8348efdab697cf31f0c9d0d2b8715c6e658f749d42a0f6ec5529f"}
Sep 16 18:41:22.012375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:22.012051    2562 scope.go:117] "RemoveContainer" containerID="5c2da46da0d7f49f539ad030d8eb2ca3cad74bedf13264d2fbdb5861446ee706"
Sep 16 18:41:22.022052 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:22.022031    2562 scope.go:117] "RemoveContainer" containerID="5c2da46da0d7f49f539ad030d8eb2ca3cad74bedf13264d2fbdb5861446ee706"
Sep 16 18:41:22.022323 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:41:22.022304    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2da46da0d7f49f539ad030d8eb2ca3cad74bedf13264d2fbdb5861446ee706\": container with ID starting with 5c2da46da0d7f49f539ad030d8eb2ca3cad74bedf13264d2fbdb5861446ee706 not found: ID does not exist" containerID="5c2da46da0d7f49f539ad030d8eb2ca3cad74bedf13264d2fbdb5861446ee706"
Sep 16 18:41:22.022396 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:22.022335    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2da46da0d7f49f539ad030d8eb2ca3cad74bedf13264d2fbdb5861446ee706"} err="failed to get container status \"5c2da46da0d7f49f539ad030d8eb2ca3cad74bedf13264d2fbdb5861446ee706\": rpc error: code = NotFound desc = could not find container \"5c2da46da0d7f49f539ad030d8eb2ca3cad74bedf13264d2fbdb5861446ee706\": container with ID starting with 5c2da46da0d7f49f539ad030d8eb2ca3cad74bedf13264d2fbdb5861446ee706 not found: ID does not exist"
Sep 16 18:41:22.050463 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:22.050431    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"]
Sep 16 18:41:22.058343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:22.058319    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-c8b23c/noten-model-c8b23c-kserve-68d9db66ff-tpz5g"]
Sep 16 18:41:22.344348 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:22.344264    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:22.551533 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:22.551500    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="793d66ce-acdd-4042-8cea-1b4f64f01e37" path="/var/lib/kubelet/pods/793d66ce-acdd-4042-8cea-1b4f64f01e37/volumes"
Sep 16 18:41:22.551992 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:22.551975    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e429196-411e-4152-9084-d9f5d9af7a40" path="/var/lib/kubelet/pods/9e429196-411e-4152-9084-d9f5d9af7a40/volumes"
Sep 16 18:41:23.850446 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:23.850383    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:24.047184 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:24.047141    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:24.950926 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:24.950874    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:25.174555 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:25.174465    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:25.999520 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:25.999470    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:27.761159 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:27.761119    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:29.435300 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:29.435266    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:29.635249 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:29.635217    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:30.004774 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:30.004733    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:30.189220 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:30.189183    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:30.584132 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:30.584099    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"]
Sep 16 18:41:30.584650 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:30.584365    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np" podUID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerName="istio-proxy" containerID="cri-o://8a156642174c1c1c234eb8b39c9f200af3ab9ab89df86e69c6cf2aaff298efee" gracePeriod=30
Sep 16 18:41:30.625045 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:30.625015    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:31.097803 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.097765    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb"]
Sep 16 18:41:31.098229 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.098206    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="793d66ce-acdd-4042-8cea-1b4f64f01e37" containerName="main"
Sep 16 18:41:31.098229 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.098227    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="793d66ce-acdd-4042-8cea-1b4f64f01e37" containerName="main"
Sep 16 18:41:31.098403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.098245    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e429196-411e-4152-9084-d9f5d9af7a40" containerName="main"
Sep 16 18:41:31.098403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.098251    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e429196-411e-4152-9084-d9f5d9af7a40" containerName="main"
Sep 16 18:41:31.098403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.098267    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c7d397-a627-4768-b2ef-42824aceae16" containerName="main"
Sep 16 18:41:31.098403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.098273    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c7d397-a627-4768-b2ef-42824aceae16" containerName="main"
Sep 16 18:41:31.098403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.098337    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e429196-411e-4152-9084-d9f5d9af7a40" containerName="main"
Sep 16 18:41:31.098403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.098348    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="05c7d397-a627-4768-b2ef-42824aceae16" containerName="main"
Sep 16 18:41:31.098403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.098354    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="793d66ce-acdd-4042-8cea-1b4f64f01e37" containerName="main"
Sep 16 18:41:31.102862 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.102843    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb"
Sep 16 18:41:31.131857 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.131830    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:41:31.138087 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.138068    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:41:31.157335 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.157310    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb"]
Sep 16 18:41:31.213018 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.212990    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de4daa3-2c0a-4142-a1a3-949aed9e931c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb\" (UID: \"5de4daa3-2c0a-4142-a1a3-949aed9e931c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb"
Sep 16 18:41:31.313749 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.313714    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de4daa3-2c0a-4142-a1a3-949aed9e931c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb\" (UID: \"5de4daa3-2c0a-4142-a1a3-949aed9e931c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb"
Sep 16 18:41:31.314324 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.314303    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de4daa3-2c0a-4142-a1a3-949aed9e931c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb\" (UID: \"5de4daa3-2c0a-4142-a1a3-949aed9e931c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb"
Sep 16 18:41:31.413516 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.412963    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb"
Sep 16 18:41:31.499963 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.499907    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:31.734722 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:31.734686    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb"]
Sep 16 18:41:31.736062 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:41:31.736036    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de4daa3_2c0a_4142_a1a3_949aed9e931c.slice/crio-c2d12e3cb175835a5c62800857a2e049d30514d58a9a4c9d1ed66e1220317a9e WatchSource:0}: Error finding container c2d12e3cb175835a5c62800857a2e049d30514d58a9a4c9d1ed66e1220317a9e: Status 404 returned error can't find the container with id c2d12e3cb175835a5c62800857a2e049d30514d58a9a4c9d1ed66e1220317a9e
Sep 16 18:41:32.081183 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:32.081151    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb" event={"ID":"5de4daa3-2c0a-4142-a1a3-949aed9e931c","Type":"ContainerStarted","Data":"f7e33790d22e55a9f5c2c1a402f75e936187e6cc468cc0ce5bfc93570cecbc07"}
Sep 16 18:41:32.081356 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:32.081204    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb" event={"ID":"5de4daa3-2c0a-4142-a1a3-949aed9e931c","Type":"ContainerStarted","Data":"c2d12e3cb175835a5c62800857a2e049d30514d58a9a4c9d1ed66e1220317a9e"}
Sep 16 18:41:32.125999 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:32.125947    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb" podStartSLOduration=1.125931322 podStartE2EDuration="1.125931322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:32.125700386 +0000 UTC m=+1529.990604434" watchObservedRunningTime="2026-09-16 18:41:32.125931322 +0000 UTC m=+1529.990835369"
Sep 16 18:41:33.089954 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:33.089915    2562 generic.go:356] "Generic (PLEG): container finished" podID="5de4daa3-2c0a-4142-a1a3-949aed9e931c" containerID="f7e33790d22e55a9f5c2c1a402f75e936187e6cc468cc0ce5bfc93570cecbc07" exitCode=0
Sep 16 18:41:33.090361 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:33.089990    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb" event={"ID":"5de4daa3-2c0a-4142-a1a3-949aed9e931c","Type":"ContainerDied","Data":"f7e33790d22e55a9f5c2c1a402f75e936187e6cc468cc0ce5bfc93570cecbc07"}
Sep 16 18:41:33.182656 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:33.182623    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:33.870653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:33.870619    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:41:34.245286 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:34.245262    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb"
Sep 16 18:41:34.343462 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:34.343431    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de4daa3-2c0a-4142-a1a3-949aed9e931c-maas-api-service-ca\") pod \"5de4daa3-2c0a-4142-a1a3-949aed9e931c\" (UID: \"5de4daa3-2c0a-4142-a1a3-949aed9e931c\") "
Sep 16 18:41:34.343831 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:34.343806    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de4daa3-2c0a-4142-a1a3-949aed9e931c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5de4daa3-2c0a-4142-a1a3-949aed9e931c" (UID: "5de4daa3-2c0a-4142-a1a3-949aed9e931c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:41:34.444269 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:34.444189    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5de4daa3-2c0a-4142-a1a3-949aed9e931c-maas-api-service-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:34.597470 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:34.597394    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:35.104946 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.104909    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb" event={"ID":"5de4daa3-2c0a-4142-a1a3-949aed9e931c","Type":"ContainerDied","Data":"c2d12e3cb175835a5c62800857a2e049d30514d58a9a4c9d1ed66e1220317a9e"}
Sep 16 18:41:35.105124 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.104967    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2d12e3cb175835a5c62800857a2e049d30514d58a9a4c9d1ed66e1220317a9e"
Sep 16 18:41:35.105124 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.104919    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb"
Sep 16 18:41:35.839636 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.839607    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:41:35.959267 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.959237    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-token\") pod \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") "
Sep 16 18:41:35.959433 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.959329    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-credential-socket\") pod \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") "
Sep 16 18:41:35.959477 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.959436    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cspk\" (UniqueName: \"kubernetes.io/projected/04c9e55b-1b9f-4eda-866b-a9060c4eb871-kube-api-access-5cspk\") pod \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") "
Sep 16 18:41:35.959524 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.959492    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-workload-certs\") pod \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") "
Sep 16 18:41:35.959609 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.959579    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-podinfo\") pod \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") "
Sep 16 18:41:35.959652 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.959611    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-data\") pod \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") "
Sep 16 18:41:35.959652 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.959637    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-envoy\") pod \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") "
Sep 16 18:41:35.959753 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.959641    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04c9e55b-1b9f-4eda-866b-a9060c4eb871" (UID: "04c9e55b-1b9f-4eda-866b-a9060c4eb871"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:35.959753 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.959673    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istiod-ca-cert\") pod \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") "
Sep 16 18:41:35.959753 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.959700    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-workload-socket\") pod \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\" (UID: \"04c9e55b-1b9f-4eda-866b-a9060c4eb871\") "
Sep 16 18:41:35.959919 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.959753    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04c9e55b-1b9f-4eda-866b-a9060c4eb871" (UID: "04c9e55b-1b9f-4eda-866b-a9060c4eb871"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:35.959919 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.959832    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-data" (OuterVolumeSpecName: "istio-data") pod "04c9e55b-1b9f-4eda-866b-a9060c4eb871" (UID: "04c9e55b-1b9f-4eda-866b-a9060c4eb871"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:35.960146 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.960122    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04c9e55b-1b9f-4eda-866b-a9060c4eb871" (UID: "04c9e55b-1b9f-4eda-866b-a9060c4eb871"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:41:35.960263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.960146    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-data\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:35.960263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.960168    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-credential-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:35.960263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.960183    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-workload-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:35.960427 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.960356    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04c9e55b-1b9f-4eda-866b-a9060c4eb871" (UID: "04c9e55b-1b9f-4eda-866b-a9060c4eb871"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:35.962144 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.962116    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-token" (OuterVolumeSpecName: "istio-token") pod "04c9e55b-1b9f-4eda-866b-a9060c4eb871" (UID: "04c9e55b-1b9f-4eda-866b-a9060c4eb871"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:35.962278 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.962221    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c9e55b-1b9f-4eda-866b-a9060c4eb871-kube-api-access-5cspk" (OuterVolumeSpecName: "kube-api-access-5cspk") pod "04c9e55b-1b9f-4eda-866b-a9060c4eb871" (UID: "04c9e55b-1b9f-4eda-866b-a9060c4eb871"). InnerVolumeSpecName "kube-api-access-5cspk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:35.962352 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.962299    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04c9e55b-1b9f-4eda-866b-a9060c4eb871" (UID: "04c9e55b-1b9f-4eda-866b-a9060c4eb871"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:35.962475 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:35.962454    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04c9e55b-1b9f-4eda-866b-a9060c4eb871" (UID: "04c9e55b-1b9f-4eda-866b-a9060c4eb871"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:41:36.061218 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.061176    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-podinfo\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:36.061218 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.061217    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-envoy\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:36.061758 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.061231    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istiod-ca-cert\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:36.061758 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.061244    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04c9e55b-1b9f-4eda-866b-a9060c4eb871-workload-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:36.061758 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.061258    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04c9e55b-1b9f-4eda-866b-a9060c4eb871-istio-token\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:36.061758 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.061274    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cspk\" (UniqueName: \"kubernetes.io/projected/04c9e55b-1b9f-4eda-866b-a9060c4eb871-kube-api-access-5cspk\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:41:36.066255 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.066223    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:36.112775 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.112743    2562 generic.go:356] "Generic (PLEG): container finished" podID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerID="8a156642174c1c1c234eb8b39c9f200af3ab9ab89df86e69c6cf2aaff298efee" exitCode=0
Sep 16 18:41:36.112943 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.112794    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"
Sep 16 18:41:36.112943 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.112830    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np" event={"ID":"04c9e55b-1b9f-4eda-866b-a9060c4eb871","Type":"ContainerDied","Data":"8a156642174c1c1c234eb8b39c9f200af3ab9ab89df86e69c6cf2aaff298efee"}
Sep 16 18:41:36.112943 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.112884    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np" event={"ID":"04c9e55b-1b9f-4eda-866b-a9060c4eb871","Type":"ContainerDied","Data":"00391a7e3c10bc13e76dbb390c84565668747632d68cee5f32ee43f70f6a7321"}
Sep 16 18:41:36.112943 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.112912    2562 scope.go:117] "RemoveContainer" containerID="8a156642174c1c1c234eb8b39c9f200af3ab9ab89df86e69c6cf2aaff298efee"
Sep 16 18:41:36.123903 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.123883    2562 scope.go:117] "RemoveContainer" containerID="09e927d77dac8671838d16aaeae9ebe7ffc62964fd0cba6a6ddb0cbe0fea53cf"
Sep 16 18:41:36.132671 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.132652    2562 scope.go:117] "RemoveContainer" containerID="8a156642174c1c1c234eb8b39c9f200af3ab9ab89df86e69c6cf2aaff298efee"
Sep 16 18:41:36.132933 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:41:36.132915    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a156642174c1c1c234eb8b39c9f200af3ab9ab89df86e69c6cf2aaff298efee\": container with ID starting with 8a156642174c1c1c234eb8b39c9f200af3ab9ab89df86e69c6cf2aaff298efee not found: ID does not exist" containerID="8a156642174c1c1c234eb8b39c9f200af3ab9ab89df86e69c6cf2aaff298efee"
Sep 16 18:41:36.132984 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.132942    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a156642174c1c1c234eb8b39c9f200af3ab9ab89df86e69c6cf2aaff298efee"} err="failed to get container status \"8a156642174c1c1c234eb8b39c9f200af3ab9ab89df86e69c6cf2aaff298efee\": rpc error: code = NotFound desc = could not find container \"8a156642174c1c1c234eb8b39c9f200af3ab9ab89df86e69c6cf2aaff298efee\": container with ID starting with 8a156642174c1c1c234eb8b39c9f200af3ab9ab89df86e69c6cf2aaff298efee not found: ID does not exist"
Sep 16 18:41:36.132984 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.132959    2562 scope.go:117] "RemoveContainer" containerID="09e927d77dac8671838d16aaeae9ebe7ffc62964fd0cba6a6ddb0cbe0fea53cf"
Sep 16 18:41:36.133234 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:41:36.133213    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e927d77dac8671838d16aaeae9ebe7ffc62964fd0cba6a6ddb0cbe0fea53cf\": container with ID starting with 09e927d77dac8671838d16aaeae9ebe7ffc62964fd0cba6a6ddb0cbe0fea53cf not found: ID does not exist" containerID="09e927d77dac8671838d16aaeae9ebe7ffc62964fd0cba6a6ddb0cbe0fea53cf"
Sep 16 18:41:36.133283 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.133239    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e927d77dac8671838d16aaeae9ebe7ffc62964fd0cba6a6ddb0cbe0fea53cf"} err="failed to get container status \"09e927d77dac8671838d16aaeae9ebe7ffc62964fd0cba6a6ddb0cbe0fea53cf\": rpc error: code = NotFound desc = could not find container \"09e927d77dac8671838d16aaeae9ebe7ffc62964fd0cba6a6ddb0cbe0fea53cf\": container with ID starting with 09e927d77dac8671838d16aaeae9ebe7ffc62964fd0cba6a6ddb0cbe0fea53cf not found: ID does not exist"
Sep 16 18:41:36.157954 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.157928    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"]
Sep 16 18:41:36.161101 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.161081    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4ab744-openshift-default-66fdc6978b-t49np"]
Sep 16 18:41:36.551872 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:36.551833    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" path="/var/lib/kubelet/pods/04c9e55b-1b9f-4eda-866b-a9060c4eb871/volumes"
Sep 16 18:41:37.361407 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:37.361374    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:37.621862 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:37.621780    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:37.993364 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:37.993331    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:38.577229 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.577197    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"]
Sep 16 18:41:38.577627 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.577612    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5de4daa3-2c0a-4142-a1a3-949aed9e931c" containerName="revoke-keys"
Sep 16 18:41:38.577672 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.577634    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5de4daa3-2c0a-4142-a1a3-949aed9e931c" containerName="revoke-keys"
Sep 16 18:41:38.577672 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.577651    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerName="istio-proxy"
Sep 16 18:41:38.577672 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.577657    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerName="istio-proxy"
Sep 16 18:41:38.577672 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.577666    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerName="istio-proxy"
Sep 16 18:41:38.577672 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.577672    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerName="istio-proxy"
Sep 16 18:41:38.577879 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.577757    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5de4daa3-2c0a-4142-a1a3-949aed9e931c" containerName="revoke-keys"
Sep 16 18:41:38.577879 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.577765    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerName="istio-proxy"
Sep 16 18:41:38.577879 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.577773    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="04c9e55b-1b9f-4eda-866b-a9060c4eb871" containerName="istio-proxy"
Sep 16 18:41:38.582294 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.582277    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.589889 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.589863    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-520999-openshift-default-dockercfg-rwl7j\""
Sep 16 18:41:38.641413 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.641379    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"]
Sep 16 18:41:38.678437 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.678410    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:38.688888 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.688716    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-workload-socket\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.688888 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.688767    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-workload-certs\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.688888 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.688805    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-podinfo\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.688888 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.688840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-token\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.688888 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.688871    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istiod-ca-cert\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.688888 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.688895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-envoy\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.689337 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.688928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-data\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.689337 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.688970    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ttr\" (UniqueName: \"kubernetes.io/projected/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-kube-api-access-v6ttr\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.689337 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.689045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-credential-socket\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.790111 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.790068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-credential-socket\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.790282 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.790156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-workload-socket\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.790282 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.790177    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-workload-certs\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.790282 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.790196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-podinfo\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.790282 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.790219    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-token\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.790282 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.790246    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istiod-ca-cert\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.790282 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.790270    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-envoy\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.790674 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.790302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-data\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.790674 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.790328    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ttr\" (UniqueName: \"kubernetes.io/projected/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-kube-api-access-v6ttr\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.790674 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.790515    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-credential-socket\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.790674 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.790627    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-workload-socket\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.790833 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.790733    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-data\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.790926 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.790904    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-workload-certs\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.791152 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.791126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istiod-ca-cert\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.792658 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.792635    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-envoy\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.792782 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.792765    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-podinfo\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.834085 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.834017    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ttr\" (UniqueName: \"kubernetes.io/projected/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-kube-api-access-v6ttr\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.844156 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.844132    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-token\") pod \"e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") " pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.892330 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.892290    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:38.978585 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:38.978525    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:39.212892 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:39.212862    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"]
Sep 16 18:41:39.215460 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:41:39.215430    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2d03b1_40e6_4025_9c8e_f63bc2ce9de6.slice/crio-0f6c4b93a392a5b8cbd04986734cfca2b1b403702d864fdb56422a5dd64adccb WatchSource:0}: Error finding container 0f6c4b93a392a5b8cbd04986734cfca2b1b403702d864fdb56422a5dd64adccb: Status 404 returned error can't find the container with id 0f6c4b93a392a5b8cbd04986734cfca2b1b403702d864fdb56422a5dd64adccb
Sep 16 18:41:39.217676 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:39.217644    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:41:39.217776 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:39.217716    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:41:39.217776 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:39.217746    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:41:39.569577 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:39.569466    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:40.141681 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:40.141632    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx" event={"ID":"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6","Type":"ContainerStarted","Data":"56e059e23d05a7f981ced60a5e017f8554d92416ade4e684ad0671970627e1a2"}
Sep 16 18:41:40.142165 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:40.141707    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx" event={"ID":"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6","Type":"ContainerStarted","Data":"0f6c4b93a392a5b8cbd04986734cfca2b1b403702d864fdb56422a5dd64adccb"}
Sep 16 18:41:40.189861 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:40.189806    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx" podStartSLOduration=2.189789293 podStartE2EDuration="2.189789293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:40.187961764 +0000 UTC m=+1538.052865825" watchObservedRunningTime="2026-09-16 18:41:40.189789293 +0000 UTC m=+1538.054693341"
Sep 16 18:41:40.892728 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:40.892685    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:40.897701 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:40.897669    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:41.051720 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:41.051683    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:41.147967 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:41.147880    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:41.148805 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:41.148783    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:41:42.442673 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:42.442635    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:42.654509 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:42.654473    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:43.096528 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:43.095940    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:44.419308 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:44.419271    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:45.319441 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:45.319402    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:45.485567 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:45.485512    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:45.610782 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:45.610684    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:45.767126 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:45.767092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:45.798339 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:45.798306    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb"]
Sep 16 18:41:45.821925 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:45.821897    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6ef84e-fjscb"]
Sep 16 18:41:45.952375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:45.952339    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:46.198708 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:46.198669    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:46.357497 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:46.357400    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:46.551261 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:46.551228    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5de4daa3-2c0a-4142-a1a3-949aed9e931c" path="/var/lib/kubelet/pods/5de4daa3-2c0a-4142-a1a3-949aed9e931c/volumes"
Sep 16 18:41:46.757206 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:46.757172    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:47.051572 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:47.051473    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:47.823476 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:47.823441    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.428323 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:49.428292    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:51.423716 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:51.423682    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:52.822577 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:52.822528    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"]
Sep 16 18:41:52.827346 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:52.827321    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:52.831962 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:52.831935    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d8a600\"/\"e2e-conflict-b-8b493098-kserve-self-signed-certs\""
Sep 16 18:41:52.866714 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:52.866681    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"]
Sep 16 18:41:52.931281 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:52.931245    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-home\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:52.931478 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:52.931295    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-tmp-dir\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:52.931478 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:52.931378    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeb69706-c123-4d2f-ac05-fa3438b2952e-tls-certs\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:52.931478 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:52.931429    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-model-cache\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:52.931657 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:52.931493    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4kjl\" (UniqueName: \"kubernetes.io/projected/aeb69706-c123-4d2f-ac05-fa3438b2952e-kube-api-access-q4kjl\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:52.931657 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:52.931602    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-dshm\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:53.032571 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.032507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeb69706-c123-4d2f-ac05-fa3438b2952e-tls-certs\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:53.032769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.032600    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-model-cache\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:53.032769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.032629    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4kjl\" (UniqueName: \"kubernetes.io/projected/aeb69706-c123-4d2f-ac05-fa3438b2952e-kube-api-access-q4kjl\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:53.032769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.032674    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-dshm\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:53.032769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.032756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-home\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:53.032999 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.032787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-tmp-dir\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:53.033106 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.033081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-model-cache\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:53.033169 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.033129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-tmp-dir\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:53.033218 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.033167    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-home\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:53.034881 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.034860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-dshm\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:53.035140 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.035122    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeb69706-c123-4d2f-ac05-fa3438b2952e-tls-certs\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:53.060237 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.060205    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4kjl\" (UniqueName: \"kubernetes.io/projected/aeb69706-c123-4d2f-ac05-fa3438b2952e-kube-api-access-q4kjl\") pod \"e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") " pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:53.064125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.064080    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:53.136941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.136846    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:41:53.175194 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.175159    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:53.440275 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:53.440250    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"]
Sep 16 18:41:53.442156 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:41:53.442128    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaeb69706_c123_4d2f_ac05_fa3438b2952e.slice/crio-c05baf877907cace277b86df8b0c07344dc795c9e72a4d0c30b12e01a9d1d867 WatchSource:0}: Error finding container c05baf877907cace277b86df8b0c07344dc795c9e72a4d0c30b12e01a9d1d867: Status 404 returned error can't find the container with id c05baf877907cace277b86df8b0c07344dc795c9e72a4d0c30b12e01a9d1d867
Sep 16 18:41:54.235538 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:54.235500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb" event={"ID":"aeb69706-c123-4d2f-ac05-fa3438b2952e","Type":"ContainerStarted","Data":"9bb0ee65e02ffddf57cc6a5374109d83b088029e21ec483a2c1ee9f6af4f16ee"}
Sep 16 18:41:54.235976 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:54.235580    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb" event={"ID":"aeb69706-c123-4d2f-ac05-fa3438b2952e","Type":"ContainerStarted","Data":"c05baf877907cace277b86df8b0c07344dc795c9e72a4d0c30b12e01a9d1d867"}
Sep 16 18:41:54.298843 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:54.298786    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb" podStartSLOduration=2.298770079 podStartE2EDuration="2.298770079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:54.2969915 +0000 UTC m=+1552.161895547" watchObservedRunningTime="2026-09-16 18:41:54.298770079 +0000 UTC m=+1552.163674160"
Sep 16 18:41:54.674977 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:54.674884    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:55.158808 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:55.158769    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:56.526434 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:56.526400    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:56.669224 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:56.669187    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:56.807922 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:56.807836    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:56.852233 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:56.852189    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"]
Sep 16 18:41:56.852426 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:56.852399    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz" podUID="befa9b40-d175-440e-b9c5-ff809620b697" containerName="istio-proxy" containerID="cri-o://5a4e2d9fae6f038fb6ddcccb29ee1ad725ae432cea9aba92ec07b8762a0f8c51" gracePeriod=30
Sep 16 18:41:56.951996 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:56.951943    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:57.091622 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:57.091462    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:57.503820 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:57.503778    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:57.889263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:57.889188    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7873-5bee0d"]
Sep 16 18:41:57.892916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:57.892893    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7873-5bee0d"
Sep 16 18:41:57.901078 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:57.901058    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fd5lk\""
Sep 16 18:41:57.920692 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:57.920665    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7873-5bee0d"]
Sep 16 18:41:57.987009 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:57.984309    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c4sn\" (UniqueName: \"kubernetes.io/projected/7fcbbb21-de7e-4c2e-9d66-e37f32356354-kube-api-access-5c4sn\") pod \"test-curl-7873-5bee0d\" (UID: \"7fcbbb21-de7e-4c2e-9d66-e37f32356354\") " pod="opendatahub/test-curl-7873-5bee0d"
Sep 16 18:41:58.019255 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:58.019220    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:58.085322 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:58.085291    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c4sn\" (UniqueName: \"kubernetes.io/projected/7fcbbb21-de7e-4c2e-9d66-e37f32356354-kube-api-access-5c4sn\") pod \"test-curl-7873-5bee0d\" (UID: \"7fcbbb21-de7e-4c2e-9d66-e37f32356354\") " pod="opendatahub/test-curl-7873-5bee0d"
Sep 16 18:41:58.177538 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:58.177465    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c4sn\" (UniqueName: \"kubernetes.io/projected/7fcbbb21-de7e-4c2e-9d66-e37f32356354-kube-api-access-5c4sn\") pod \"test-curl-7873-5bee0d\" (UID: \"7fcbbb21-de7e-4c2e-9d66-e37f32356354\") " pod="opendatahub/test-curl-7873-5bee0d"
Sep 16 18:41:58.202437 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:58.202400    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7873-5bee0d"
Sep 16 18:41:58.362384 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:58.362341    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:58.499853 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:58.499812    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7873-5bee0d"]
Sep 16 18:41:58.501414 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:41:58.501388    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fcbbb21_de7e_4c2e_9d66_e37f32356354.slice/crio-f7125fbfdbdbc31d8e6e28b206f1d5aa9d02463a39c710baddcceba821812797 WatchSource:0}: Error finding container f7125fbfdbdbc31d8e6e28b206f1d5aa9d02463a39c710baddcceba821812797: Status 404 returned error can't find the container with id f7125fbfdbdbc31d8e6e28b206f1d5aa9d02463a39c710baddcceba821812797
Sep 16 18:41:58.668778 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:58.668748    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:59.273098 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:59.273063    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7873-5bee0d" event={"ID":"7fcbbb21-de7e-4c2e-9d66-e37f32356354","Type":"ContainerStarted","Data":"da1e9ab8832f20311e1a4ea37017ee1ca6f169470ab6d4f0c594e47c109eb858"}
Sep 16 18:41:59.273501 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:59.273106    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7873-5bee0d" event={"ID":"7fcbbb21-de7e-4c2e-9d66-e37f32356354","Type":"ContainerStarted","Data":"f7125fbfdbdbc31d8e6e28b206f1d5aa9d02463a39c710baddcceba821812797"}
Sep 16 18:41:59.568269 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:59.568182    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:59.946694 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:59.946579    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7873-5bee0d" podStartSLOduration=2.946555397 podStartE2EDuration="2.946555397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:59.291028565 +0000 UTC m=+1557.155932612" watchObservedRunningTime="2026-09-16 18:41:59.946555397 +0000 UTC m=+1557.811459437"
Sep 16 18:41:59.947496 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:59.947463    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7873-5bee0d"]
Sep 16 18:41:59.954585 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:41:59.954343    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7873-5bee0d"]
Sep 16 18:42:00.279915 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:42:00.279882    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7873-5bee0d" container="test-curl-7873-5bee0d"
Sep 16 18:42:00.557068 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:00.556976    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:01.286681 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:01.286620    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7873-5bee0d" podUID="7fcbbb21-de7e-4c2e-9d66-e37f32356354" containerName="test-curl-7873-5bee0d" containerID="cri-o://da1e9ab8832f20311e1a4ea37017ee1ca6f169470ab6d4f0c594e47c109eb858" gracePeriod=2
Sep 16 18:42:01.363679 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:01.363640    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:01.489082 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:01.489042    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:01.668043 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:01.667946    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:02.115480 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.115454    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:42:02.227639 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.227533    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-credential-socket\") pod \"befa9b40-d175-440e-b9c5-ff809620b697\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") "
Sep 16 18:42:02.227639 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.227599    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befa9b40-d175-440e-b9c5-ff809620b697-istio-token\") pod \"befa9b40-d175-440e-b9c5-ff809620b697\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") "
Sep 16 18:42:02.227866 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.227652    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-workload-certs\") pod \"befa9b40-d175-440e-b9c5-ff809620b697\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") "
Sep 16 18:42:02.227866 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.227709    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-istio-data\") pod \"befa9b40-d175-440e-b9c5-ff809620b697\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") "
Sep 16 18:42:02.227866 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.227745    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-workload-socket\") pod \"befa9b40-d175-440e-b9c5-ff809620b697\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") "
Sep 16 18:42:02.227866 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.227779    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befa9b40-d175-440e-b9c5-ff809620b697-istio-podinfo\") pod \"befa9b40-d175-440e-b9c5-ff809620b697\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") "
Sep 16 18:42:02.227866 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.227842    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "befa9b40-d175-440e-b9c5-ff809620b697" (UID: "befa9b40-d175-440e-b9c5-ff809620b697"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:02.228126 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.227877    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befa9b40-d175-440e-b9c5-ff809620b697-istiod-ca-cert\") pod \"befa9b40-d175-440e-b9c5-ff809620b697\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") "
Sep 16 18:42:02.228126 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.227921    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-istio-envoy\") pod \"befa9b40-d175-440e-b9c5-ff809620b697\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") "
Sep 16 18:42:02.228126 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.227949    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2mwc\" (UniqueName: \"kubernetes.io/projected/befa9b40-d175-440e-b9c5-ff809620b697-kube-api-access-s2mwc\") pod \"befa9b40-d175-440e-b9c5-ff809620b697\" (UID: \"befa9b40-d175-440e-b9c5-ff809620b697\") "
Sep 16 18:42:02.228126 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.228030    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-istio-data" (OuterVolumeSpecName: "istio-data") pod "befa9b40-d175-440e-b9c5-ff809620b697" (UID: "befa9b40-d175-440e-b9c5-ff809620b697"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:02.228126 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.228060    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "befa9b40-d175-440e-b9c5-ff809620b697" (UID: "befa9b40-d175-440e-b9c5-ff809620b697"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:02.228399 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.228330    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/befa9b40-d175-440e-b9c5-ff809620b697-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "befa9b40-d175-440e-b9c5-ff809620b697" (UID: "befa9b40-d175-440e-b9c5-ff809620b697"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:02.228457 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.228402    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-credential-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:42:02.228457 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.228416    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "befa9b40-d175-440e-b9c5-ff809620b697" (UID: "befa9b40-d175-440e-b9c5-ff809620b697"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:02.228457 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.228424    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-workload-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:42:02.228664 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.228462    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-istio-data\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:42:02.230339 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.230311    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/befa9b40-d175-440e-b9c5-ff809620b697-istio-token" (OuterVolumeSpecName: "istio-token") pod "befa9b40-d175-440e-b9c5-ff809620b697" (UID: "befa9b40-d175-440e-b9c5-ff809620b697"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:02.230482 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.230360    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/befa9b40-d175-440e-b9c5-ff809620b697-kube-api-access-s2mwc" (OuterVolumeSpecName: "kube-api-access-s2mwc") pod "befa9b40-d175-440e-b9c5-ff809620b697" (UID: "befa9b40-d175-440e-b9c5-ff809620b697"). InnerVolumeSpecName "kube-api-access-s2mwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:02.230538 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.230502    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "befa9b40-d175-440e-b9c5-ff809620b697" (UID: "befa9b40-d175-440e-b9c5-ff809620b697"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:02.230656 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.230626    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/befa9b40-d175-440e-b9c5-ff809620b697-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "befa9b40-d175-440e-b9c5-ff809620b697" (UID: "befa9b40-d175-440e-b9c5-ff809620b697"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:42:02.294421 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.294385    2562 generic.go:356] "Generic (PLEG): container finished" podID="befa9b40-d175-440e-b9c5-ff809620b697" containerID="5a4e2d9fae6f038fb6ddcccb29ee1ad725ae432cea9aba92ec07b8762a0f8c51" exitCode=0
Sep 16 18:42:02.294852 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.294432    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"
Sep 16 18:42:02.294852 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.294470    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz" event={"ID":"befa9b40-d175-440e-b9c5-ff809620b697","Type":"ContainerDied","Data":"5a4e2d9fae6f038fb6ddcccb29ee1ad725ae432cea9aba92ec07b8762a0f8c51"}
Sep 16 18:42:02.294852 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.294527    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz" event={"ID":"befa9b40-d175-440e-b9c5-ff809620b697","Type":"ContainerDied","Data":"684d302511836dc4bf28093a7d56218aafce9785856b6eae4b384fc3b6ddf6ae"}
Sep 16 18:42:02.294852 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.294577    2562 scope.go:117] "RemoveContainer" containerID="5a4e2d9fae6f038fb6ddcccb29ee1ad725ae432cea9aba92ec07b8762a0f8c51"
Sep 16 18:42:02.306134 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.306117    2562 scope.go:117] "RemoveContainer" containerID="a373737b3f649b401fcf52529fe465e0f731c22f680631ba0bac72a054c41291"
Sep 16 18:42:02.314730 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.314709    2562 scope.go:117] "RemoveContainer" containerID="5a4e2d9fae6f038fb6ddcccb29ee1ad725ae432cea9aba92ec07b8762a0f8c51"
Sep 16 18:42:02.314991 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:42:02.314973    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a4e2d9fae6f038fb6ddcccb29ee1ad725ae432cea9aba92ec07b8762a0f8c51\": container with ID starting with 5a4e2d9fae6f038fb6ddcccb29ee1ad725ae432cea9aba92ec07b8762a0f8c51 not found: ID does not exist" containerID="5a4e2d9fae6f038fb6ddcccb29ee1ad725ae432cea9aba92ec07b8762a0f8c51"
Sep 16 18:42:02.315034 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.315000    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a4e2d9fae6f038fb6ddcccb29ee1ad725ae432cea9aba92ec07b8762a0f8c51"} err="failed to get container status \"5a4e2d9fae6f038fb6ddcccb29ee1ad725ae432cea9aba92ec07b8762a0f8c51\": rpc error: code = NotFound desc = could not find container \"5a4e2d9fae6f038fb6ddcccb29ee1ad725ae432cea9aba92ec07b8762a0f8c51\": container with ID starting with 5a4e2d9fae6f038fb6ddcccb29ee1ad725ae432cea9aba92ec07b8762a0f8c51 not found: ID does not exist"
Sep 16 18:42:02.315034 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.315018    2562 scope.go:117] "RemoveContainer" containerID="a373737b3f649b401fcf52529fe465e0f731c22f680631ba0bac72a054c41291"
Sep 16 18:42:02.315310 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:42:02.315284    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a373737b3f649b401fcf52529fe465e0f731c22f680631ba0bac72a054c41291\": container with ID starting with a373737b3f649b401fcf52529fe465e0f731c22f680631ba0bac72a054c41291 not found: ID does not exist" containerID="a373737b3f649b401fcf52529fe465e0f731c22f680631ba0bac72a054c41291"
Sep 16 18:42:02.315378 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.315320    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a373737b3f649b401fcf52529fe465e0f731c22f680631ba0bac72a054c41291"} err="failed to get container status \"a373737b3f649b401fcf52529fe465e0f731c22f680631ba0bac72a054c41291\": rpc error: code = NotFound desc = could not find container \"a373737b3f649b401fcf52529fe465e0f731c22f680631ba0bac72a054c41291\": container with ID starting with a373737b3f649b401fcf52529fe465e0f731c22f680631ba0bac72a054c41291 not found: ID does not exist"
Sep 16 18:42:02.319474 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.319451    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"]
Sep 16 18:42:02.323151 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.323131    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6ef84e-openshift-default-857958fcbd-gwmnz"]
Sep 16 18:42:02.329986 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.329966    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befa9b40-d175-440e-b9c5-ff809620b697-istio-token\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:42:02.330055 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.329988    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-workload-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:42:02.330055 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.329997    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befa9b40-d175-440e-b9c5-ff809620b697-istio-podinfo\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:42:02.330055 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.330006    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befa9b40-d175-440e-b9c5-ff809620b697-istiod-ca-cert\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:42:02.330055 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.330014    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befa9b40-d175-440e-b9c5-ff809620b697-istio-envoy\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:42:02.330055 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.330021    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2mwc\" (UniqueName: \"kubernetes.io/projected/befa9b40-d175-440e-b9c5-ff809620b697-kube-api-access-s2mwc\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:42:02.462937 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.462900    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7873-ace0cd"]
Sep 16 18:42:02.463462 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.463443    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fcbbb21-de7e-4c2e-9d66-e37f32356354" containerName="test-curl-7873-5bee0d"
Sep 16 18:42:02.463507 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.463465    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fcbbb21-de7e-4c2e-9d66-e37f32356354" containerName="test-curl-7873-5bee0d"
Sep 16 18:42:02.463507 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.463487    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="befa9b40-d175-440e-b9c5-ff809620b697" containerName="istio-proxy"
Sep 16 18:42:02.463507 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.463493    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="befa9b40-d175-440e-b9c5-ff809620b697" containerName="istio-proxy"
Sep 16 18:42:02.463639 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.463507    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="befa9b40-d175-440e-b9c5-ff809620b697" containerName="istio-proxy"
Sep 16 18:42:02.463639 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.463513    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="befa9b40-d175-440e-b9c5-ff809620b697" containerName="istio-proxy"
Sep 16 18:42:02.463639 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.463609    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="befa9b40-d175-440e-b9c5-ff809620b697" containerName="istio-proxy"
Sep 16 18:42:02.463639 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.463621    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="befa9b40-d175-440e-b9c5-ff809620b697" containerName="istio-proxy"
Sep 16 18:42:02.463639 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.463628    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fcbbb21-de7e-4c2e-9d66-e37f32356354" containerName="test-curl-7873-5bee0d"
Sep 16 18:42:02.468245 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.468222    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7873-ace0cd"
Sep 16 18:42:02.474788 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.474759    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7873-ace0cd"]
Sep 16 18:42:02.532000 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.531917    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfffj\" (UniqueName: \"kubernetes.io/projected/1161f4b5-b636-42d9-9004-3ee9c35cc745-kube-api-access-nfffj\") pod \"test-curl-7873-ace0cd\" (UID: \"1161f4b5-b636-42d9-9004-3ee9c35cc745\") " pod="opendatahub/test-curl-7873-ace0cd"
Sep 16 18:42:02.559619 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.559051    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="befa9b40-d175-440e-b9c5-ff809620b697" path="/var/lib/kubelet/pods/befa9b40-d175-440e-b9c5-ff809620b697/volumes"
Sep 16 18:42:02.615730 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.615700    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:02.633054 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.633016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfffj\" (UniqueName: \"kubernetes.io/projected/1161f4b5-b636-42d9-9004-3ee9c35cc745-kube-api-access-nfffj\") pod \"test-curl-7873-ace0cd\" (UID: \"1161f4b5-b636-42d9-9004-3ee9c35cc745\") " pod="opendatahub/test-curl-7873-ace0cd"
Sep 16 18:42:02.650522 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.650486    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfffj\" (UniqueName: \"kubernetes.io/projected/1161f4b5-b636-42d9-9004-3ee9c35cc745-kube-api-access-nfffj\") pod \"test-curl-7873-ace0cd\" (UID: \"1161f4b5-b636-42d9-9004-3ee9c35cc745\") " pod="opendatahub/test-curl-7873-ace0cd"
Sep 16 18:42:02.716780 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.716744    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:02.774675 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.774434    2562 scope.go:117] "RemoveContainer" containerID="b9326e066fe656a666b74c80bb3c2557348fcd3c1e0d61fc12e07fcef1fabd4f"
Sep 16 18:42:02.778504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.778475    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7873-ace0cd"
Sep 16 18:42:02.824970 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:02.824933    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:03.058845 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:03.058769    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7873-ace0cd"]
Sep 16 18:42:03.062274 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:42:03.062245    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1161f4b5_b636_42d9_9004_3ee9c35cc745.slice/crio-04dc52e8ae649592786469b0992952fdf466e7869462de19c7a424f33dc63549 WatchSource:0}: Error finding container 04dc52e8ae649592786469b0992952fdf466e7869462de19c7a424f33dc63549: Status 404 returned error can't find the container with id 04dc52e8ae649592786469b0992952fdf466e7869462de19c7a424f33dc63549
Sep 16 18:42:03.137471 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:03.137445    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:42:03.137626 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:03.137494    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:42:03.151641 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:03.151614    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:42:03.306053 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:03.305830    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:03.306653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:03.306624    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7873-ace0cd" event={"ID":"1161f4b5-b636-42d9-9004-3ee9c35cc745","Type":"ContainerStarted","Data":"0df7b60ec55be29653d96a16fde62de06161878a90f1e56fbd323a1eef17cf83"}
Sep 16 18:42:03.306777 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:03.306672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7873-ace0cd" event={"ID":"1161f4b5-b636-42d9-9004-3ee9c35cc745","Type":"ContainerStarted","Data":"04dc52e8ae649592786469b0992952fdf466e7869462de19c7a424f33dc63549"}
Sep 16 18:42:03.323155 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:03.323049    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:42:03.340918 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:03.340869    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7873-ace0cd" podStartSLOduration=1.340854107 podStartE2EDuration="1.340854107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:03.335969725 +0000 UTC m=+1561.200873774" watchObservedRunningTime="2026-09-16 18:42:03.340854107 +0000 UTC m=+1561.205758155"
Sep 16 18:42:03.444992 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:03.444965    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7873-5bee0d"
Sep 16 18:42:03.449989 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:03.449947    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:03.542593 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:03.542533    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c4sn\" (UniqueName: \"kubernetes.io/projected/7fcbbb21-de7e-4c2e-9d66-e37f32356354-kube-api-access-5c4sn\") pod \"7fcbbb21-de7e-4c2e-9d66-e37f32356354\" (UID: \"7fcbbb21-de7e-4c2e-9d66-e37f32356354\") "
Sep 16 18:42:03.544594 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:03.544561    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fcbbb21-de7e-4c2e-9d66-e37f32356354-kube-api-access-5c4sn" (OuterVolumeSpecName: "kube-api-access-5c4sn") pod "7fcbbb21-de7e-4c2e-9d66-e37f32356354" (UID: "7fcbbb21-de7e-4c2e-9d66-e37f32356354"). InnerVolumeSpecName "kube-api-access-5c4sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:03.644258 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:03.644186    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c4sn\" (UniqueName: \"kubernetes.io/projected/7fcbbb21-de7e-4c2e-9d66-e37f32356354-kube-api-access-5c4sn\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:42:04.012814 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.012778    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:04.199958 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.199905    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7873-ace0cd"]
Sep 16 18:42:04.208476 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.208437    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7873-ace0cd"]
Sep 16 18:42:04.248576 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.248495    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:42:04.316843 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.316750    2562 generic.go:356] "Generic (PLEG): container finished" podID="7fcbbb21-de7e-4c2e-9d66-e37f32356354" containerID="da1e9ab8832f20311e1a4ea37017ee1ca6f169470ab6d4f0c594e47c109eb858" exitCode=137
Sep 16 18:42:04.316843 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.316798    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7873-5bee0d"
Sep 16 18:42:04.317338 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.316853    2562 scope.go:117] "RemoveContainer" containerID="da1e9ab8832f20311e1a4ea37017ee1ca6f169470ab6d4f0c594e47c109eb858"
Sep 16 18:42:04.317338 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:42:04.317325    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7873-ace0cd" container="test-curl-7873-ace0cd"
Sep 16 18:42:04.327231 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.327208    2562 scope.go:117] "RemoveContainer" containerID="da1e9ab8832f20311e1a4ea37017ee1ca6f169470ab6d4f0c594e47c109eb858"
Sep 16 18:42:04.327563 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:42:04.327523    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1e9ab8832f20311e1a4ea37017ee1ca6f169470ab6d4f0c594e47c109eb858\": container with ID starting with da1e9ab8832f20311e1a4ea37017ee1ca6f169470ab6d4f0c594e47c109eb858 not found: ID does not exist" containerID="da1e9ab8832f20311e1a4ea37017ee1ca6f169470ab6d4f0c594e47c109eb858"
Sep 16 18:42:04.327647 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.327575    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1e9ab8832f20311e1a4ea37017ee1ca6f169470ab6d4f0c594e47c109eb858"} err="failed to get container status \"da1e9ab8832f20311e1a4ea37017ee1ca6f169470ab6d4f0c594e47c109eb858\": rpc error: code = NotFound desc = could not find container \"da1e9ab8832f20311e1a4ea37017ee1ca6f169470ab6d4f0c594e47c109eb858\": container with ID starting with da1e9ab8832f20311e1a4ea37017ee1ca6f169470ab6d4f0c594e47c109eb858 not found: ID does not exist"
Sep 16 18:42:04.329273 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:42:04.329246    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7873-5bee0d\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" podUID="7fcbbb21-de7e-4c2e-9d66-e37f32356354" pod="opendatahub/test-curl-7873-5bee0d"
Sep 16 18:42:04.348924 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.348886    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7873-0d03d2"]
Sep 16 18:42:04.349527 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.349504    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1161f4b5-b636-42d9-9004-3ee9c35cc745" containerName="test-curl-7873-ace0cd"
Sep 16 18:42:04.349618 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.349533    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1161f4b5-b636-42d9-9004-3ee9c35cc745" containerName="test-curl-7873-ace0cd"
Sep 16 18:42:04.349721 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.349682    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1161f4b5-b636-42d9-9004-3ee9c35cc745" containerName="test-curl-7873-ace0cd"
Sep 16 18:42:04.354791 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.354765    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7873-0d03d2"
Sep 16 18:42:04.357658 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:42:04.357624    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7873-5bee0d\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" podUID="7fcbbb21-de7e-4c2e-9d66-e37f32356354" pod="opendatahub/test-curl-7873-5bee0d"
Sep 16 18:42:04.376792 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.376763    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7873-0d03d2"]
Sep 16 18:42:04.452188 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.451941    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8xkc\" (UniqueName: \"kubernetes.io/projected/34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe-kube-api-access-v8xkc\") pod \"test-curl-7873-0d03d2\" (UID: \"34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe\") " pod="opendatahub/test-curl-7873-0d03d2"
Sep 16 18:42:04.552832 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.552796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8xkc\" (UniqueName: \"kubernetes.io/projected/34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe-kube-api-access-v8xkc\") pod \"test-curl-7873-0d03d2\" (UID: \"34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe\") " pod="opendatahub/test-curl-7873-0d03d2"
Sep 16 18:42:04.554503 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.554471    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fcbbb21-de7e-4c2e-9d66-e37f32356354" path="/var/lib/kubelet/pods/7fcbbb21-de7e-4c2e-9d66-e37f32356354/volumes"
Sep 16 18:42:04.588028 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.587955    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8xkc\" (UniqueName: \"kubernetes.io/projected/34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe-kube-api-access-v8xkc\") pod \"test-curl-7873-0d03d2\" (UID: \"34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe\") " pod="opendatahub/test-curl-7873-0d03d2"
Sep 16 18:42:04.665355 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.665321    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7873-0d03d2"
Sep 16 18:42:04.994480 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:42:04.994449    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34cd8422_b2ba_4bdb_ba4d_2aa7d3c7b1fe.slice/crio-8ded44d1c98e5a6c66d8201b24b92f5984927b8ece5a6470c637cfb02ee49535 WatchSource:0}: Error finding container 8ded44d1c98e5a6c66d8201b24b92f5984927b8ece5a6470c637cfb02ee49535: Status 404 returned error can't find the container with id 8ded44d1c98e5a6c66d8201b24b92f5984927b8ece5a6470c637cfb02ee49535
Sep 16 18:42:04.995337 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:04.995285    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7873-0d03d2"]
Sep 16 18:42:05.326851 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:05.326739    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7873-0d03d2" event={"ID":"34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe","Type":"ContainerStarted","Data":"28b737d3c73cea84f9706395267caf7032ab1592a06e3504d7c3b8dbbf155851"}
Sep 16 18:42:05.326851 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:05.326789    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7873-0d03d2" event={"ID":"34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe","Type":"ContainerStarted","Data":"8ded44d1c98e5a6c66d8201b24b92f5984927b8ece5a6470c637cfb02ee49535"}
Sep 16 18:42:05.328098 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:05.328067    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7873-ace0cd" podUID="1161f4b5-b636-42d9-9004-3ee9c35cc745" containerName="test-curl-7873-ace0cd" containerID="cri-o://0df7b60ec55be29653d96a16fde62de06161878a90f1e56fbd323a1eef17cf83" gracePeriod=2
Sep 16 18:42:06.418718 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:06.418601    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7873-0d03d2" podStartSLOduration=2.418581153 podStartE2EDuration="2.418581153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:05.447280406 +0000 UTC m=+1563.312184454" watchObservedRunningTime="2026-09-16 18:42:06.418581153 +0000 UTC m=+1564.283485247"
Sep 16 18:42:06.434766 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:06.433850    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7873-0d03d2"]
Sep 16 18:42:06.438373 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:06.438192    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7873-0d03d2"]
Sep 16 18:42:06.526560 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:06.526508    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:07.127512 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.127478    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:07.342865 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.342830    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7873-0d03d2" podUID="34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe" containerName="test-curl-7873-0d03d2" containerID="cri-o://28b737d3c73cea84f9706395267caf7032ab1592a06e3504d7c3b8dbbf155851" gracePeriod=2
Sep 16 18:42:07.564681 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.564609    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"]
Sep 16 18:42:07.565984 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.565955    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe" containerName="test-curl-7873-0d03d2"
Sep 16 18:42:07.566166 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.566153    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe" containerName="test-curl-7873-0d03d2"
Sep 16 18:42:07.566439 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.566426    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe" containerName="test-curl-7873-0d03d2"
Sep 16 18:42:07.580495 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.579801    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.583409 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.583377    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ab83f76b-openshift-default-dockercfg-2jwdh\""
Sep 16 18:42:07.585734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.585699    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7873-ace0cd"
Sep 16 18:42:07.590632 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.590605    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"]
Sep 16 18:42:07.618850 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.618810    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:07.691116 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.689963    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfffj\" (UniqueName: \"kubernetes.io/projected/1161f4b5-b636-42d9-9004-3ee9c35cc745-kube-api-access-nfffj\") pod \"1161f4b5-b636-42d9-9004-3ee9c35cc745\" (UID: \"1161f4b5-b636-42d9-9004-3ee9c35cc745\") "
Sep 16 18:42:07.691116 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.690184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6kx6\" (UniqueName: \"kubernetes.io/projected/df4a86ae-3905-4322-8108-c6780fd87ac8-kube-api-access-r6kx6\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.691116 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.690311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-token\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.691116 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.690340    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-credential-socket\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.691116 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.690405    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-data\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.691116 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.690430    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-podinfo\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.691116 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.690480    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4a86ae-3905-4322-8108-c6780fd87ac8-istiod-ca-cert\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.691116 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.690559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-workload-certs\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.691116 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.690591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-envoy\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.691116 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.690638    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-workload-socket\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.695870 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.695782    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1161f4b5-b636-42d9-9004-3ee9c35cc745-kube-api-access-nfffj" (OuterVolumeSpecName: "kube-api-access-nfffj") pod "1161f4b5-b636-42d9-9004-3ee9c35cc745" (UID: "1161f4b5-b636-42d9-9004-3ee9c35cc745"). InnerVolumeSpecName "kube-api-access-nfffj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:07.791943 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.791851    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-workload-socket\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.791943 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.791906    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6kx6\" (UniqueName: \"kubernetes.io/projected/df4a86ae-3905-4322-8108-c6780fd87ac8-kube-api-access-r6kx6\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.792181 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.791954    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-token\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.792181 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.791973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-credential-socket\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.792181 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.791998    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-data\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.792181 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.792015    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-podinfo\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.792181 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.792052    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4a86ae-3905-4322-8108-c6780fd87ac8-istiod-ca-cert\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.792181 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.792101    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-workload-certs\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.792181 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.792130    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-envoy\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.792181 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.792180    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfffj\" (UniqueName: \"kubernetes.io/projected/1161f4b5-b636-42d9-9004-3ee9c35cc745-kube-api-access-nfffj\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:42:07.792627 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.792398    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-workload-socket\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.792627 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.792454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-credential-socket\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.792627 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.792509    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-data\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.792769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.792744    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-workload-certs\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.792913 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.792896    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4a86ae-3905-4322-8108-c6780fd87ac8-istiod-ca-cert\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.794412 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.794392    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-envoy\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.794675 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.794656    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-podinfo\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.810113 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.810078    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-token\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.810252 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.810149    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6kx6\" (UniqueName: \"kubernetes.io/projected/df4a86ae-3905-4322-8108-c6780fd87ac8-kube-api-access-r6kx6\") pod \"e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") " pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:07.833912 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.833876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:07.895657 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:07.895625    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:08.204888 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:08.204840    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:08.222438 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:08.222413    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"]
Sep 16 18:42:08.225232 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:42:08.225198    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4a86ae_3905_4322_8108_c6780fd87ac8.slice/crio-a83b814bb5c0edc5de088b7732f28068407783d4781bb885bec9f4cb3143e3ca WatchSource:0}: Error finding container a83b814bb5c0edc5de088b7732f28068407783d4781bb885bec9f4cb3143e3ca: Status 404 returned error can't find the container with id a83b814bb5c0edc5de088b7732f28068407783d4781bb885bec9f4cb3143e3ca
Sep 16 18:42:08.229632 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:08.229590    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:42:08.229713 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:08.229669    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:42:08.229713 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:08.229704    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:42:08.356572 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:08.353163    2562 generic.go:356] "Generic (PLEG): container finished" podID="1161f4b5-b636-42d9-9004-3ee9c35cc745" containerID="0df7b60ec55be29653d96a16fde62de06161878a90f1e56fbd323a1eef17cf83" exitCode=137
Sep 16 18:42:08.356572 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:08.353355    2562 scope.go:117] "RemoveContainer" containerID="0df7b60ec55be29653d96a16fde62de06161878a90f1e56fbd323a1eef17cf83"
Sep 16 18:42:08.356572 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:08.353497    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7873-ace0cd"
Sep 16 18:42:08.363665 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:08.362652    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb" event={"ID":"df4a86ae-3905-4322-8108-c6780fd87ac8","Type":"ContainerStarted","Data":"406c16239f7c8de63ee9736f237786e62c92d50b73055a69b2fae925b68d82bb"}
Sep 16 18:42:08.363665 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:08.362714    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb" event={"ID":"df4a86ae-3905-4322-8108-c6780fd87ac8","Type":"ContainerStarted","Data":"a83b814bb5c0edc5de088b7732f28068407783d4781bb885bec9f4cb3143e3ca"}
Sep 16 18:42:08.388333 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:08.388307    2562 scope.go:117] "RemoveContainer" containerID="0df7b60ec55be29653d96a16fde62de06161878a90f1e56fbd323a1eef17cf83"
Sep 16 18:42:08.389098 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:42:08.388939    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df7b60ec55be29653d96a16fde62de06161878a90f1e56fbd323a1eef17cf83\": container with ID starting with 0df7b60ec55be29653d96a16fde62de06161878a90f1e56fbd323a1eef17cf83 not found: ID does not exist" containerID="0df7b60ec55be29653d96a16fde62de06161878a90f1e56fbd323a1eef17cf83"
Sep 16 18:42:08.389098 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:08.388979    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df7b60ec55be29653d96a16fde62de06161878a90f1e56fbd323a1eef17cf83"} err="failed to get container status \"0df7b60ec55be29653d96a16fde62de06161878a90f1e56fbd323a1eef17cf83\": rpc error: code = NotFound desc = could not find container \"0df7b60ec55be29653d96a16fde62de06161878a90f1e56fbd323a1eef17cf83\": container with ID starting with 0df7b60ec55be29653d96a16fde62de06161878a90f1e56fbd323a1eef17cf83 not found: ID does not exist"
Sep 16 18:42:08.401464 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:08.401417    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb" podStartSLOduration=1.401401054 podStartE2EDuration="1.401401054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:08.399781724 +0000 UTC m=+1566.264685782" watchObservedRunningTime="2026-09-16 18:42:08.401401054 +0000 UTC m=+1566.266305102"
Sep 16 18:42:08.403508 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:42:08.403477    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7873-ace0cd\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" podUID="1161f4b5-b636-42d9-9004-3ee9c35cc745" pod="opendatahub/test-curl-7873-ace0cd"
Sep 16 18:42:08.555671 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:08.555580    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1161f4b5-b636-42d9-9004-3ee9c35cc745" path="/var/lib/kubelet/pods/1161f4b5-b636-42d9-9004-3ee9c35cc745/volumes"
Sep 16 18:42:08.896477 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:08.896388    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:09.435561 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:09.435519    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:09.584214 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:09.584180    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7873-0d03d2"
Sep 16 18:42:09.709757 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:09.709720    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8xkc\" (UniqueName: \"kubernetes.io/projected/34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe-kube-api-access-v8xkc\") pod \"34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe\" (UID: \"34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe\") "
Sep 16 18:42:09.711865 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:09.711838    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe-kube-api-access-v8xkc" (OuterVolumeSpecName: "kube-api-access-v8xkc") pod "34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe" (UID: "34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe"). InnerVolumeSpecName "kube-api-access-v8xkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:09.810969 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:09.810931    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8xkc\" (UniqueName: \"kubernetes.io/projected/34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe-kube-api-access-v8xkc\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:42:09.901079 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:09.901046    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:10.390504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:10.390467    2562 generic.go:356] "Generic (PLEG): container finished" podID="34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe" containerID="28b737d3c73cea84f9706395267caf7032ab1592a06e3504d7c3b8dbbf155851" exitCode=137
Sep 16 18:42:10.390695 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:10.390521    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7873-0d03d2"
Sep 16 18:42:10.390695 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:10.390569    2562 scope.go:117] "RemoveContainer" containerID="28b737d3c73cea84f9706395267caf7032ab1592a06e3504d7c3b8dbbf155851"
Sep 16 18:42:10.390915 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:10.390899    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:10.392095 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:10.392077    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:10.400369 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:10.400353    2562 scope.go:117] "RemoveContainer" containerID="28b737d3c73cea84f9706395267caf7032ab1592a06e3504d7c3b8dbbf155851"
Sep 16 18:42:10.400717 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:42:10.400698    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28b737d3c73cea84f9706395267caf7032ab1592a06e3504d7c3b8dbbf155851\": container with ID starting with 28b737d3c73cea84f9706395267caf7032ab1592a06e3504d7c3b8dbbf155851 not found: ID does not exist" containerID="28b737d3c73cea84f9706395267caf7032ab1592a06e3504d7c3b8dbbf155851"
Sep 16 18:42:10.400769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:10.400725    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b737d3c73cea84f9706395267caf7032ab1592a06e3504d7c3b8dbbf155851"} err="failed to get container status \"28b737d3c73cea84f9706395267caf7032ab1592a06e3504d7c3b8dbbf155851\": rpc error: code = NotFound desc = could not find container \"28b737d3c73cea84f9706395267caf7032ab1592a06e3504d7c3b8dbbf155851\": container with ID starting with 28b737d3c73cea84f9706395267caf7032ab1592a06e3504d7c3b8dbbf155851 not found: ID does not exist"
Sep 16 18:42:10.434197 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:42:10.434159    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7873-0d03d2\" is forbidden: User \"system:node:ip-10-0-142-48.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-48.ec2.internal' and this object" podUID="34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe" pod="opendatahub/test-curl-7873-0d03d2"
Sep 16 18:42:10.563568 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:10.562996    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe" path="/var/lib/kubelet/pods/34cd8422-b2ba-4bdb-ba4d-2aa7d3c7b1fe/volumes"
Sep 16 18:42:10.687121 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:10.687037    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:10.826749 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:10.826712    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:11.185374 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:11.185343    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:11.975059 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:11.975023    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:13.114373 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:13.114335    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:13.215101 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:13.215065    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:13.714330 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:13.714287    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:13.827321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:13.827278    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:14.273592 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:14.273554    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:15.385220 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:15.385179    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:15.430335 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:15.430280    2562 generic.go:356] "Generic (PLEG): container finished" podID="fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" containerID="56e059e23d05a7f981ced60a5e017f8554d92416ade4e684ad0671970627e1a2" exitCode=0
Sep 16 18:42:15.430514 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:15.430475    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx" event={"ID":"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6","Type":"ContainerDied","Data":"56e059e23d05a7f981ced60a5e017f8554d92416ade4e684ad0671970627e1a2"}
Sep 16 18:42:15.431208 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:15.431176    2562 scope.go:117] "RemoveContainer" containerID="56e059e23d05a7f981ced60a5e017f8554d92416ade4e684ad0671970627e1a2"
Sep 16 18:42:15.433002 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:15.432954    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:42:15.433237 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:15.433062    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:42:15.433237 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:15.433110    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:42:16.137101 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:16.137071    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:16.345985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:16.345951    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:16.439118 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:16.439017    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx" event={"ID":"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6","Type":"ContainerStarted","Data":"32ac34a6faa390a81ac1442063270fd9b4b9ca46105973ed9e17f771257493a6"}
Sep 16 18:42:16.516180 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:16.516147    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:16.893188 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:16.893154    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:42:16.893364 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:16.893310    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 16 18:42:16.893414 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:16.893366    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx" podUID="fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 16 18:42:17.023579 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.023521    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"]
Sep 16 18:42:17.026314 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.026293    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.031687 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.031665    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ee3af0-openshift-default-dockercfg-m8wvz\""
Sep 16 18:42:17.064445 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.064405    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:17.102237 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.102201    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"]
Sep 16 18:42:17.187966 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.187873    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-workload-socket\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.187966 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.187932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-workload-certs\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.188148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.188007    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-token\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.188148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.188110    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-envoy\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.188222 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.188170    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-data\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.188222 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.188204    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-credential-socket\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.188222 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.188219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istiod-ca-cert\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.188335 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.188245    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phmr6\" (UniqueName: \"kubernetes.io/projected/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-kube-api-access-phmr6\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.188335 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.188290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-podinfo\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.289213 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.289173    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-workload-socket\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.289425 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.289228    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-workload-certs\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.289425 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.289255    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-token\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.289425 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.289293    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-envoy\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.289425 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.289333    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-data\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.289715 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.289474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-credential-socket\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.289715 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.289519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istiod-ca-cert\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.289715 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.289567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phmr6\" (UniqueName: \"kubernetes.io/projected/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-kube-api-access-phmr6\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.289715 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.289622    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-podinfo\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.289715 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.289679    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-workload-socket\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.289997 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.289716    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-data\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.289997 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.289774    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-workload-certs\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.289997 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.289873    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-credential-socket\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.290332 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.290308    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istiod-ca-cert\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.291950 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.291924    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-envoy\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.292108 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.292089    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-podinfo\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.326477 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.326445    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phmr6\" (UniqueName: \"kubernetes.io/projected/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-kube-api-access-phmr6\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.335442 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.335407    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:17.357582 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.353075    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-token\") pod \"e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") " pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.637066 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.637030    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:17.892699 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.892613    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 16 18:42:17.892699 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.892670    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx" podUID="fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 16 18:42:17.980849 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.980729    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"]
Sep 16 18:42:17.983693 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:42:17.983664    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9450ef_fcd8_467d_9be6_28b3d67d79f8.slice/crio-55dc0fdc88a0a48e56f29d723fb3222f900326cb00c904360a91d5d41892f8c1 WatchSource:0}: Error finding container 55dc0fdc88a0a48e56f29d723fb3222f900326cb00c904360a91d5d41892f8c1: Status 404 returned error can't find the container with id 55dc0fdc88a0a48e56f29d723fb3222f900326cb00c904360a91d5d41892f8c1
Sep 16 18:42:17.986216 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.986184    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:42:17.986286 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.986252    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:42:17.986286 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:17.986281    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:42:18.282569 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:18.281780    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:18.458813 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:18.458767    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952" event={"ID":"4d9450ef-fcd8-467d-9be6-28b3d67d79f8","Type":"ContainerStarted","Data":"8ef5146b7963eaafb69ae0ccafe0a02c5ec94aec4b9bd73497f2c6c552f0093a"}
Sep 16 18:42:18.459026 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:18.458842    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952" event={"ID":"4d9450ef-fcd8-467d-9be6-28b3d67d79f8","Type":"ContainerStarted","Data":"55dc0fdc88a0a48e56f29d723fb3222f900326cb00c904360a91d5d41892f8c1"}
Sep 16 18:42:18.551454 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:18.551338    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952" podStartSLOduration=2.5513144309999998 podStartE2EDuration="2.551314431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:18.546606934 +0000 UTC m=+1576.411510980" watchObservedRunningTime="2026-09-16 18:42:18.551314431 +0000 UTC m=+1576.416218485"
Sep 16 18:42:18.638138 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:18.638102    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:18.892817 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:18.892726    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 16 18:42:18.892817 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:18.892785    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx" podUID="fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 16 18:42:19.643211 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:19.643169    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:19.893716 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:19.893622    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 16 18:42:19.893716 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:19.893693    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx" podUID="fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 16 18:42:20.088321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:20.088292    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:20.481935 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:20.481901    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:20.483577 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:20.483531    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:20.892697 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:20.892615    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Sep 16 18:42:20.892697 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:20.892671    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx" podUID="fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Sep 16 18:42:20.911114 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:20.911077    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:21.015263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:21.015233    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:21.379970 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:21.379932    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:21.897064 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:21.897036    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:42:21.897451 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:21.897327    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:42:21.898220 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:21.898201    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:42:22.861926 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:22.861886    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:23.950711 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:23.950664    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:24.059592 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:24.059539    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:24.383474 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:24.383388    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:24.649865 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:24.649740    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:24.746531 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:24.746474    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:25.333494 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:25.333457    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:25.605499 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:25.605397    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:26.651498 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:26.651457    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:27.682166 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:27.682127    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:28.605103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:28.605072    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:29.624429 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:29.624398    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:30.855925 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:30.855887    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:31.803404 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:31.803372    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:32.566116 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:32.566073    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:34.068852 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:34.068813    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:34.877706 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:34.877673    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:35.449578 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:35.449519    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:36.037064 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:36.037036    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:36.622934 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:36.622901    2562 generic.go:356] "Generic (PLEG): container finished" podID="df4a86ae-3905-4322-8108-c6780fd87ac8" containerID="406c16239f7c8de63ee9736f237786e62c92d50b73055a69b2fae925b68d82bb" exitCode=0
Sep 16 18:42:36.623324 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:36.622974    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb" event={"ID":"df4a86ae-3905-4322-8108-c6780fd87ac8","Type":"ContainerDied","Data":"406c16239f7c8de63ee9736f237786e62c92d50b73055a69b2fae925b68d82bb"}
Sep 16 18:42:36.623404 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:36.623390    2562 scope.go:117] "RemoveContainer" containerID="406c16239f7c8de63ee9736f237786e62c92d50b73055a69b2fae925b68d82bb"
Sep 16 18:42:36.624706 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:36.624679    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:42:36.624783 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:36.624767    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:42:36.624824 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:36.624798    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:42:36.859772 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:36.859730    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:37.631427 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:37.631389    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb" event={"ID":"df4a86ae-3905-4322-8108-c6780fd87ac8","Type":"ContainerStarted","Data":"73badb4f85f1770ea5fab00a95559840b6a113fa155c2522ad7c3a2d696e9fb9"}
Sep 16 18:42:37.806332 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:37.806303    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:37.895841 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:37.895757    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:37.896892 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:37.896867    2562 patch_prober.go:28] interesting pod/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 16 18:42:37.896959 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:37.896918    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb" podUID="df4a86ae-3905-4322-8108-c6780fd87ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 16 18:42:38.877520 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:38.877384    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:38.896343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:38.896310    2562 patch_prober.go:28] interesting pod/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 16 18:42:38.896783 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:38.896366    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb" podUID="df4a86ae-3905-4322-8108-c6780fd87ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 16 18:42:39.875834 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:39.875786    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:39.896035 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:39.895996    2562 patch_prober.go:28] interesting pod/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 16 18:42:39.896436 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:39.896062    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb" podUID="df4a86ae-3905-4322-8108-c6780fd87ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 16 18:42:40.390909 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:40.390864    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:40.896225 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:40.896182    2562 patch_prober.go:28] interesting pod/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 16 18:42:40.896662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:40.896247    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb" podUID="df4a86ae-3905-4322-8108-c6780fd87ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 16 18:42:41.135913 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:41.135879    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:41.896026 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:41.895989    2562 patch_prober.go:28] interesting pod/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 16 18:42:41.896209 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:41.896048    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb" podUID="df4a86ae-3905-4322-8108-c6780fd87ac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 16 18:42:42.900926 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:42.900895    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:42.902025 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:42.902003    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:42:46.557370 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:46.557330    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:42:57.772883 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:57.772850    2562 generic.go:356] "Generic (PLEG): container finished" podID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" containerID="8ef5146b7963eaafb69ae0ccafe0a02c5ec94aec4b9bd73497f2c6c552f0093a" exitCode=0
Sep 16 18:42:57.773300 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:57.772925    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952" event={"ID":"4d9450ef-fcd8-467d-9be6-28b3d67d79f8","Type":"ContainerDied","Data":"8ef5146b7963eaafb69ae0ccafe0a02c5ec94aec4b9bd73497f2c6c552f0093a"}
Sep 16 18:42:57.773361 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:57.773345    2562 scope.go:117] "RemoveContainer" containerID="8ef5146b7963eaafb69ae0ccafe0a02c5ec94aec4b9bd73497f2c6c552f0093a"
Sep 16 18:42:57.774653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:57.774628    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:42:57.774741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:57.774708    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:42:57.774779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:57.774740    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:42:58.784946 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:58.784909    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952" event={"ID":"4d9450ef-fcd8-467d-9be6-28b3d67d79f8","Type":"ContainerStarted","Data":"d5f19ce291c1c6c3499974ab1b1d25cedbca0c4f77ec8da40261fdfab2b45a11"}
Sep 16 18:42:59.637898 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:59.637854    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:42:59.638162 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:59.638135    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 16 18:42:59.638267 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:42:59.638197    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952" podUID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 16 18:43:00.638373 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:00.638332    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 16 18:43:00.638852 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:00.638400    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952" podUID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 16 18:43:01.638332 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:01.638285    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 16 18:43:01.638526 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:01.638361    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952" podUID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 16 18:43:02.637897 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:02.637858    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 16 18:43:02.638096 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:02.637920    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952" podUID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 16 18:43:03.638210 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:03.638176    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Sep 16 18:43:03.638679 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:03.638653    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952" podUID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Sep 16 18:43:04.642694 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:04.642663    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:43:04.643134 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:04.642978    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:43:04.643625 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:04.643606    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:43:36.603364 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.603330    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"]
Sep 16 18:43:36.606475 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.606451    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.609162 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.609140    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-909a40\"/\"distinct-2-w0-909a40-kserve-self-signed-certs\""
Sep 16 18:43:36.616283 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.616261    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"]
Sep 16 18:43:36.630645 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.630617    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:36.748133 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.748099    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-model-cache\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.748307 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.748238    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-home\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.748307 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.748275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-dshm\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.748432 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.748369    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-tls-certs\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.748432 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.748405    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxltm\" (UniqueName: \"kubernetes.io/projected/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-kube-api-access-jxltm\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.748536 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.748462    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-tmp-dir\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.849408 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.849371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-home\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.849408 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.849415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-dshm\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.849721 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.849468    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-tls-certs\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.849721 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.849489    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxltm\" (UniqueName: \"kubernetes.io/projected/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-kube-api-access-jxltm\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.849721 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.849687    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-tmp-dir\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.849886 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.849763    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-model-cache\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.850249 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.850225    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-home\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.850417 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.850318    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-tmp-dir\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.850417 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.850329    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-model-cache\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.852391 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.852366    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-dshm\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.852391 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.852380    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-tls-certs\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.858400 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.858348    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxltm\" (UniqueName: \"kubernetes.io/projected/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-kube-api-access-jxltm\") pod \"distinct-2-w0-909a40-kserve-58db6d447-gznfk\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") " pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:36.916511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:36.916476    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:37.199997 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:37.199965    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"]
Sep 16 18:43:37.201510 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:43:37.201476    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44ac4ea4_53cc_4fa6_a799_ad93c8ee5b32.slice/crio-a29835f962f071ea72f06519fa337d2c7f60dd9d9a692c1d64835c2d130e41fb WatchSource:0}: Error finding container a29835f962f071ea72f06519fa337d2c7f60dd9d9a692c1d64835c2d130e41fb: Status 404 returned error can't find the container with id a29835f962f071ea72f06519fa337d2c7f60dd9d9a692c1d64835c2d130e41fb
Sep 16 18:43:38.059079 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:38.059039    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk" event={"ID":"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32","Type":"ContainerStarted","Data":"029d108b1a2c8f13f58a6515764e2731aa9ed17cf7465a4951eea4b1af6a3869"}
Sep 16 18:43:38.059480 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:38.059096    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk" event={"ID":"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32","Type":"ContainerStarted","Data":"a29835f962f071ea72f06519fa337d2c7f60dd9d9a692c1d64835c2d130e41fb"}
Sep 16 18:43:38.079763 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:38.079717    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk" podStartSLOduration=2.079697918 podStartE2EDuration="2.079697918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:38.077927538 +0000 UTC m=+1655.942831585" watchObservedRunningTime="2026-09-16 18:43:38.079697918 +0000 UTC m=+1655.944601966"
Sep 16 18:43:46.029482 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:46.029443    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:43:46.917581 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:46.917517    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:46.917769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:46.917606    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:46.930818 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:46.930787    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:43:47.129747 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:43:47.129715    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:44:02.858657 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:02.858621    2562 scope.go:117] "RemoveContainer" containerID="beafc65a716c1d4e1cdee0d1fc6208b3a84ed15cb39b9f35227b9e832ebf302a"
Sep 16 18:44:02.868305 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:02.868274    2562 scope.go:117] "RemoveContainer" containerID="c3b63222922f9db9004e969efa3235e59928e1f0fb744f30d82369cf72c1a13d"
Sep 16 18:44:13.972353 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:13.972313    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:14.189556 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:14.189488    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m"]
Sep 16 18:44:14.192227 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:14.192203    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m"
Sep 16 18:44:14.195847 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:14.195819    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:44:14.196199 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:14.196181    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:44:14.202945 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:14.202909    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:14.205264 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:14.205236    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m"]
Sep 16 18:44:14.251884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:14.251782    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae8a616-95a2-4499-83f0-7afbc90a1f87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m\" (UID: \"2ae8a616-95a2-4499-83f0-7afbc90a1f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m"
Sep 16 18:44:14.353568 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:14.353515    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae8a616-95a2-4499-83f0-7afbc90a1f87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m\" (UID: \"2ae8a616-95a2-4499-83f0-7afbc90a1f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m"
Sep 16 18:44:14.354191 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:14.354167    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae8a616-95a2-4499-83f0-7afbc90a1f87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m\" (UID: \"2ae8a616-95a2-4499-83f0-7afbc90a1f87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m"
Sep 16 18:44:14.375080 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:14.375044    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:14.504650 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:14.504529    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m"
Sep 16 18:44:14.799034 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:14.799004    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m"]
Sep 16 18:44:14.800243 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:44:14.800215    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae8a616_95a2_4499_83f0_7afbc90a1f87.slice/crio-e4d386d56435ddb8ec7122d546b9420d22055467d375612e44bfeb51e5b21318 WatchSource:0}: Error finding container e4d386d56435ddb8ec7122d546b9420d22055467d375612e44bfeb51e5b21318: Status 404 returned error can't find the container with id e4d386d56435ddb8ec7122d546b9420d22055467d375612e44bfeb51e5b21318
Sep 16 18:44:15.310749 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:15.310647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m" event={"ID":"2ae8a616-95a2-4499-83f0-7afbc90a1f87","Type":"ContainerStarted","Data":"bcff8b6db53a8ed15e41397a1a7858f6bdf9d8b2c4893552884916e1118cdd1d"}
Sep 16 18:44:15.311112 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:15.310754    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m" event={"ID":"2ae8a616-95a2-4499-83f0-7afbc90a1f87","Type":"ContainerStarted","Data":"e4d386d56435ddb8ec7122d546b9420d22055467d375612e44bfeb51e5b21318"}
Sep 16 18:44:15.339118 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:15.339062    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m" podStartSLOduration=1.339045733 podStartE2EDuration="1.339045733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:15.337537449 +0000 UTC m=+1693.202441498" watchObservedRunningTime="2026-09-16 18:44:15.339045733 +0000 UTC m=+1693.203949775"
Sep 16 18:44:15.421313 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:15.421274    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:16.319198 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:16.319163    2562 generic.go:356] "Generic (PLEG): container finished" podID="2ae8a616-95a2-4499-83f0-7afbc90a1f87" containerID="bcff8b6db53a8ed15e41397a1a7858f6bdf9d8b2c4893552884916e1118cdd1d" exitCode=0
Sep 16 18:44:16.319688 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:16.319218    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m" event={"ID":"2ae8a616-95a2-4499-83f0-7afbc90a1f87","Type":"ContainerDied","Data":"bcff8b6db53a8ed15e41397a1a7858f6bdf9d8b2c4893552884916e1118cdd1d"}
Sep 16 18:44:16.373819 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:16.373783    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:17.166375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:17.166333    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:17.477698 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:17.477670    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m"
Sep 16 18:44:17.591903 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:17.591859    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae8a616-95a2-4499-83f0-7afbc90a1f87-maas-api-service-ca\") pod \"2ae8a616-95a2-4499-83f0-7afbc90a1f87\" (UID: \"2ae8a616-95a2-4499-83f0-7afbc90a1f87\") "
Sep 16 18:44:17.592214 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:17.592190    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae8a616-95a2-4499-83f0-7afbc90a1f87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ae8a616-95a2-4499-83f0-7afbc90a1f87" (UID: "2ae8a616-95a2-4499-83f0-7afbc90a1f87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:17.592363 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:17.592346    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae8a616-95a2-4499-83f0-7afbc90a1f87-maas-api-service-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:18.127897 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:18.127813    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:18.333511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:18.333485    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m"
Sep 16 18:44:18.333708 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:18.333480    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m" event={"ID":"2ae8a616-95a2-4499-83f0-7afbc90a1f87","Type":"ContainerDied","Data":"e4d386d56435ddb8ec7122d546b9420d22055467d375612e44bfeb51e5b21318"}
Sep 16 18:44:18.333708 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:18.333620    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4d386d56435ddb8ec7122d546b9420d22055467d375612e44bfeb51e5b21318"
Sep 16 18:44:18.943342 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:18.943302    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"]
Sep 16 18:44:18.964604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:18.964567    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"]
Sep 16 18:44:18.976220 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:18.976182    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"]
Sep 16 18:44:19.055822 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:19.055782    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:19.148189 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:19.148149    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:20.189976 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.189925    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:20.717735 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.717701    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"]
Sep 16 18:44:20.718130 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.718117    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae8a616-95a2-4499-83f0-7afbc90a1f87" containerName="revoke-keys"
Sep 16 18:44:20.718180 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.718132    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae8a616-95a2-4499-83f0-7afbc90a1f87" containerName="revoke-keys"
Sep 16 18:44:20.718239 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.718229    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae8a616-95a2-4499-83f0-7afbc90a1f87" containerName="revoke-keys"
Sep 16 18:44:20.720262 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.720244    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.723250 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.723213    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ab83f76b\"/\"kube-root-ca.crt\""
Sep 16 18:44:20.723759 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.723740    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ab83f76b\"/\"default-dockercfg-4xfpp\""
Sep 16 18:44:20.724297 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.724276    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ab83f76b\"/\"e2e-lifecycle-model-ab83f76b-kserve-self-signed-certs\""
Sep 16 18:44:20.724403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.724375    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ab83f76b\"/\"openshift-service-ca.crt\""
Sep 16 18:44:20.736838 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.736805    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"]
Sep 16 18:44:20.754480 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.754436    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:20.826682 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.826638    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkj28\" (UniqueName: \"kubernetes.io/projected/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-kube-api-access-lkj28\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.826873 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.826702    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-home\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.826873 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.826773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-tls-certs\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.826873 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.826832    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-tmp-dir\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.826987 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.826880    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-dshm\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.826987 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.826900    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-model-cache\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.928003 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.927960    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-dshm\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.928187 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.928006    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-model-cache\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.928187 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.928056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkj28\" (UniqueName: \"kubernetes.io/projected/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-kube-api-access-lkj28\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.928187 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.928091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-home\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.928354 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.928234    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-tls-certs\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.928409 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.928349    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-tmp-dir\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.928486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.928454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-home\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.928606 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.928588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-model-cache\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.928738 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.928718    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-tmp-dir\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.930370 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.930346    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-dshm\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.930672 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.930652    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-tls-certs\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:20.944111 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:20.944082    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkj28\" (UniqueName: \"kubernetes.io/projected/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-kube-api-access-lkj28\") pod \"e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") " pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:21.030152 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:21.030049    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:21.047795 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:21.047763    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:21.331479 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:21.331447    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"]
Sep 16 18:44:21.331852 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:44:21.331760    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44ce8be2_bfbe_4b19_ad04_59f26e1a3603.slice/crio-11267c4bc6df90bbd30d86d7c5bb54f4e7b6db31a656b1dc3542f5dbc1d1013e WatchSource:0}: Error finding container 11267c4bc6df90bbd30d86d7c5bb54f4e7b6db31a656b1dc3542f5dbc1d1013e: Status 404 returned error can't find the container with id 11267c4bc6df90bbd30d86d7c5bb54f4e7b6db31a656b1dc3542f5dbc1d1013e
Sep 16 18:44:21.359969 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:21.359937    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk" event={"ID":"44ce8be2-bfbe-4b19-ad04-59f26e1a3603","Type":"ContainerStarted","Data":"11267c4bc6df90bbd30d86d7c5bb54f4e7b6db31a656b1dc3542f5dbc1d1013e"}
Sep 16 18:44:22.115046 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:22.115013    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:22.369078 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:22.368987    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk" event={"ID":"44ce8be2-bfbe-4b19-ad04-59f26e1a3603","Type":"ContainerStarted","Data":"9727d4981bc5d9d9dd54e124b6b93b5cced2cf27cdfb91e67af8a86bf5583ee1"}
Sep 16 18:44:22.397105 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:22.397047    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk" podStartSLOduration=2.3970288330000002 podStartE2EDuration="2.397028833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:22.396188583 +0000 UTC m=+1700.261092631" watchObservedRunningTime="2026-09-16 18:44:22.397028833 +0000 UTC m=+1700.261932952"
Sep 16 18:44:24.127101 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:24.127054    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:44:29.809708 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:29.809627    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:29.931430 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:29.931401    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:30.035323 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:30.035282    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m"]
Sep 16 18:44:30.039561 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:30.039509    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-909a40-m6d2m"]
Sep 16 18:44:30.058691 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:30.058657    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:30.172930 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:30.172838    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:30.400379 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:30.400340    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:30.503313 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:30.503277    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:30.552017 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:30.551972    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ae8a616-95a2-4499-83f0-7afbc90a1f87" path="/var/lib/kubelet/pods/2ae8a616-95a2-4499-83f0-7afbc90a1f87/volumes"
Sep 16 18:44:31.030309 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:31.030276    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:31.030763 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:31.030328    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:31.043313 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:31.043283    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:31.359437 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:31.359361    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:31.450155 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:31.450119    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:44:32.200776 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:32.200745    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:33.013341 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:33.013309    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:33.964710 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:33.964668    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk" podUID="44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32" containerName="main" containerID="cri-o://029d108b1a2c8f13f58a6515764e2731aa9ed17cf7465a4951eea4b1af6a3869" gracePeriod=45
Sep 16 18:44:33.967268 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:33.967234    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:33.985408 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:33.985376    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb" podUID="b0fe54a1-f24e-479d-b114-fb8bc7dd370f" containerName="main" containerID="cri-o://1034699853ad34a72cf946b69c221fff79e3da8bd0f9230dc9a6dc4bb72402d1" gracePeriod=45
Sep 16 18:44:33.996609 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:33.996529    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf" podUID="bdb505d0-3913-4ebe-b987-b80c019a2f47" containerName="main" containerID="cri-o://893e02410559310374758e3046fa5cdb25a21a0b561b9a858a3482c6a2661e14" gracePeriod=45
Sep 16 18:44:34.263113 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.263072    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:44:34.326169 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.326141    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:44:34.329496 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.329473    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:44:34.380467 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.380210    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-tmp-dir\") pod \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") "
Sep 16 18:44:34.380467 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.380258    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-home\") pod \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") "
Sep 16 18:44:34.380467 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.380307    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-model-cache\") pod \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") "
Sep 16 18:44:34.380467 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.380336    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-tls-certs\") pod \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") "
Sep 16 18:44:34.380467 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.380399    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxltm\" (UniqueName: \"kubernetes.io/projected/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-kube-api-access-jxltm\") pod \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") "
Sep 16 18:44:34.380467 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.380454    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-dshm\") pod \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\" (UID: \"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32\") "
Sep 16 18:44:34.381236 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.381065    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32" (UID: "44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:34.381236 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.381077    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-model-cache" (OuterVolumeSpecName: "model-cache") pod "44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32" (UID: "44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:34.381492 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.381441    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-home" (OuterVolumeSpecName: "home") pod "44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32" (UID: "44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:34.383690 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.383663    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32" (UID: "44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:34.383690 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.383665    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-dshm" (OuterVolumeSpecName: "dshm") pod "44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32" (UID: "44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:34.383866 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.383840    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-kube-api-access-jxltm" (OuterVolumeSpecName: "kube-api-access-jxltm") pod "44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32" (UID: "44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32"). InnerVolumeSpecName "kube-api-access-jxltm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:34.457982 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.457942    2562 generic.go:356] "Generic (PLEG): container finished" podID="44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32" containerID="029d108b1a2c8f13f58a6515764e2731aa9ed17cf7465a4951eea4b1af6a3869" exitCode=0
Sep 16 18:44:34.458175 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.458005    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"
Sep 16 18:44:34.458175 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.458035    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk" event={"ID":"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32","Type":"ContainerDied","Data":"029d108b1a2c8f13f58a6515764e2731aa9ed17cf7465a4951eea4b1af6a3869"}
Sep 16 18:44:34.458175 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.458083    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk" event={"ID":"44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32","Type":"ContainerDied","Data":"a29835f962f071ea72f06519fa337d2c7f60dd9d9a692c1d64835c2d130e41fb"}
Sep 16 18:44:34.458175 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.458105    2562 scope.go:117] "RemoveContainer" containerID="029d108b1a2c8f13f58a6515764e2731aa9ed17cf7465a4951eea4b1af6a3869"
Sep 16 18:44:34.459773 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.459745    2562 generic.go:356] "Generic (PLEG): container finished" podID="b0fe54a1-f24e-479d-b114-fb8bc7dd370f" containerID="1034699853ad34a72cf946b69c221fff79e3da8bd0f9230dc9a6dc4bb72402d1" exitCode=0
Sep 16 18:44:34.459928 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.459800    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"
Sep 16 18:44:34.459928 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.459886    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb" event={"ID":"b0fe54a1-f24e-479d-b114-fb8bc7dd370f","Type":"ContainerDied","Data":"1034699853ad34a72cf946b69c221fff79e3da8bd0f9230dc9a6dc4bb72402d1"}
Sep 16 18:44:34.459928 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.459921    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb" event={"ID":"b0fe54a1-f24e-479d-b114-fb8bc7dd370f","Type":"ContainerDied","Data":"c0ed5b230cf53ee1b19987521088f57317ec9ecdfa1a3bd94d922fc3ce383af4"}
Sep 16 18:44:34.461578 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.461552    2562 generic.go:356] "Generic (PLEG): container finished" podID="bdb505d0-3913-4ebe-b987-b80c019a2f47" containerID="893e02410559310374758e3046fa5cdb25a21a0b561b9a858a3482c6a2661e14" exitCode=0
Sep 16 18:44:34.461744 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.461584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf" event={"ID":"bdb505d0-3913-4ebe-b987-b80c019a2f47","Type":"ContainerDied","Data":"893e02410559310374758e3046fa5cdb25a21a0b561b9a858a3482c6a2661e14"}
Sep 16 18:44:34.461744 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.461614    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"
Sep 16 18:44:34.461866 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.461625    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf" event={"ID":"bdb505d0-3913-4ebe-b987-b80c019a2f47","Type":"ContainerDied","Data":"c510a924ed14bc01bbca725be50dac93ee1847b9266472c1190b2d4947acfaef"}
Sep 16 18:44:34.468883 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.468865    2562 scope.go:117] "RemoveContainer" containerID="029d108b1a2c8f13f58a6515764e2731aa9ed17cf7465a4951eea4b1af6a3869"
Sep 16 18:44:34.469151 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:44:34.469133    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"029d108b1a2c8f13f58a6515764e2731aa9ed17cf7465a4951eea4b1af6a3869\": container with ID starting with 029d108b1a2c8f13f58a6515764e2731aa9ed17cf7465a4951eea4b1af6a3869 not found: ID does not exist" containerID="029d108b1a2c8f13f58a6515764e2731aa9ed17cf7465a4951eea4b1af6a3869"
Sep 16 18:44:34.469206 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.469159    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"029d108b1a2c8f13f58a6515764e2731aa9ed17cf7465a4951eea4b1af6a3869"} err="failed to get container status \"029d108b1a2c8f13f58a6515764e2731aa9ed17cf7465a4951eea4b1af6a3869\": rpc error: code = NotFound desc = could not find container \"029d108b1a2c8f13f58a6515764e2731aa9ed17cf7465a4951eea4b1af6a3869\": container with ID starting with 029d108b1a2c8f13f58a6515764e2731aa9ed17cf7465a4951eea4b1af6a3869 not found: ID does not exist"
Sep 16 18:44:34.469206 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.469184    2562 scope.go:117] "RemoveContainer" containerID="1034699853ad34a72cf946b69c221fff79e3da8bd0f9230dc9a6dc4bb72402d1"
Sep 16 18:44:34.480175 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.480012    2562 scope.go:117] "RemoveContainer" containerID="1034699853ad34a72cf946b69c221fff79e3da8bd0f9230dc9a6dc4bb72402d1"
Sep 16 18:44:34.480337 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:44:34.480316    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1034699853ad34a72cf946b69c221fff79e3da8bd0f9230dc9a6dc4bb72402d1\": container with ID starting with 1034699853ad34a72cf946b69c221fff79e3da8bd0f9230dc9a6dc4bb72402d1 not found: ID does not exist" containerID="1034699853ad34a72cf946b69c221fff79e3da8bd0f9230dc9a6dc4bb72402d1"
Sep 16 18:44:34.480388 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.480345    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1034699853ad34a72cf946b69c221fff79e3da8bd0f9230dc9a6dc4bb72402d1"} err="failed to get container status \"1034699853ad34a72cf946b69c221fff79e3da8bd0f9230dc9a6dc4bb72402d1\": rpc error: code = NotFound desc = could not find container \"1034699853ad34a72cf946b69c221fff79e3da8bd0f9230dc9a6dc4bb72402d1\": container with ID starting with 1034699853ad34a72cf946b69c221fff79e3da8bd0f9230dc9a6dc4bb72402d1 not found: ID does not exist"
Sep 16 18:44:34.480388 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.480363    2562 scope.go:117] "RemoveContainer" containerID="893e02410559310374758e3046fa5cdb25a21a0b561b9a858a3482c6a2661e14"
Sep 16 18:44:34.481361 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.481344    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-dshm\") pod \"bdb505d0-3913-4ebe-b987-b80c019a2f47\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") "
Sep 16 18:44:34.481432 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.481394    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-tls-certs\") pod \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") "
Sep 16 18:44:34.481484 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.481450    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-home\") pod \"bdb505d0-3913-4ebe-b987-b80c019a2f47\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") "
Sep 16 18:44:34.481560 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.481532    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-tmp-dir\") pod \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") "
Sep 16 18:44:34.481630 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.481590    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-model-cache\") pod \"bdb505d0-3913-4ebe-b987-b80c019a2f47\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") "
Sep 16 18:44:34.481690 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.481625    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l9vm\" (UniqueName: \"kubernetes.io/projected/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-kube-api-access-2l9vm\") pod \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") "
Sep 16 18:44:34.481690 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.481682    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m569\" (UniqueName: \"kubernetes.io/projected/bdb505d0-3913-4ebe-b987-b80c019a2f47-kube-api-access-9m569\") pod \"bdb505d0-3913-4ebe-b987-b80c019a2f47\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") "
Sep 16 18:44:34.481798 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.481719    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-model-cache\") pod \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") "
Sep 16 18:44:34.481798 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.481746    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-dshm\") pod \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") "
Sep 16 18:44:34.481798 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.481773    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-home" (OuterVolumeSpecName: "home") pod "bdb505d0-3913-4ebe-b987-b80c019a2f47" (UID: "bdb505d0-3913-4ebe-b987-b80c019a2f47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:34.481798 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.481786    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-home\") pod \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\" (UID: \"b0fe54a1-f24e-479d-b114-fb8bc7dd370f\") "
Sep 16 18:44:34.482005 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.481807    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0fe54a1-f24e-479d-b114-fb8bc7dd370f" (UID: "b0fe54a1-f24e-479d-b114-fb8bc7dd370f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:34.482005 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.481849    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb505d0-3913-4ebe-b987-b80c019a2f47-tls-certs\") pod \"bdb505d0-3913-4ebe-b987-b80c019a2f47\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") "
Sep 16 18:44:34.482005 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.481910    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-tmp-dir\") pod \"bdb505d0-3913-4ebe-b987-b80c019a2f47\" (UID: \"bdb505d0-3913-4ebe-b987-b80c019a2f47\") "
Sep 16 18:44:34.482005 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.481999    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-home" (OuterVolumeSpecName: "home") pod "b0fe54a1-f24e-479d-b114-fb8bc7dd370f" (UID: "b0fe54a1-f24e-479d-b114-fb8bc7dd370f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:34.482225 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.482200    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0fe54a1-f24e-479d-b114-fb8bc7dd370f" (UID: "b0fe54a1-f24e-479d-b114-fb8bc7dd370f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:34.482433 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.482406    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdb505d0-3913-4ebe-b987-b80c019a2f47" (UID: "bdb505d0-3913-4ebe-b987-b80c019a2f47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:34.482510 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.482429    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.482510 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.482448    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.482510 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.482460    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxltm\" (UniqueName: \"kubernetes.io/projected/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-kube-api-access-jxltm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.482510 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.482473    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.482510 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.482486    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.482510 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.482498    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.482510 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.482510    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.483003 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.482522    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.483003 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.482533    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.483003 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.482562    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.483003 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.482659    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdb505d0-3913-4ebe-b987-b80c019a2f47" (UID: "bdb505d0-3913-4ebe-b987-b80c019a2f47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:34.484619 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.484586    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0fe54a1-f24e-479d-b114-fb8bc7dd370f" (UID: "b0fe54a1-f24e-479d-b114-fb8bc7dd370f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:34.484721 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.484694    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb505d0-3913-4ebe-b987-b80c019a2f47-kube-api-access-9m569" (OuterVolumeSpecName: "kube-api-access-9m569") pod "bdb505d0-3913-4ebe-b987-b80c019a2f47" (UID: "bdb505d0-3913-4ebe-b987-b80c019a2f47"). InnerVolumeSpecName "kube-api-access-9m569". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:34.484873 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.484824    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-dshm" (OuterVolumeSpecName: "dshm") pod "bdb505d0-3913-4ebe-b987-b80c019a2f47" (UID: "bdb505d0-3913-4ebe-b987-b80c019a2f47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:34.484873 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.484860    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-dshm" (OuterVolumeSpecName: "dshm") pod "b0fe54a1-f24e-479d-b114-fb8bc7dd370f" (UID: "b0fe54a1-f24e-479d-b114-fb8bc7dd370f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:34.485044 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.484952    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-kube-api-access-2l9vm" (OuterVolumeSpecName: "kube-api-access-2l9vm") pod "b0fe54a1-f24e-479d-b114-fb8bc7dd370f" (UID: "b0fe54a1-f24e-479d-b114-fb8bc7dd370f"). InnerVolumeSpecName "kube-api-access-2l9vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:34.485604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.485570    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb505d0-3913-4ebe-b987-b80c019a2f47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdb505d0-3913-4ebe-b987-b80c019a2f47" (UID: "bdb505d0-3913-4ebe-b987-b80c019a2f47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:34.488245 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.488223    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"]
Sep 16 18:44:34.491269 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.491248    2562 scope.go:117] "RemoveContainer" containerID="893e02410559310374758e3046fa5cdb25a21a0b561b9a858a3482c6a2661e14"
Sep 16 18:44:34.491534 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:44:34.491514    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893e02410559310374758e3046fa5cdb25a21a0b561b9a858a3482c6a2661e14\": container with ID starting with 893e02410559310374758e3046fa5cdb25a21a0b561b9a858a3482c6a2661e14 not found: ID does not exist" containerID="893e02410559310374758e3046fa5cdb25a21a0b561b9a858a3482c6a2661e14"
Sep 16 18:44:34.491624 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.491551    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893e02410559310374758e3046fa5cdb25a21a0b561b9a858a3482c6a2661e14"} err="failed to get container status \"893e02410559310374758e3046fa5cdb25a21a0b561b9a858a3482c6a2661e14\": rpc error: code = NotFound desc = could not find container \"893e02410559310374758e3046fa5cdb25a21a0b561b9a858a3482c6a2661e14\": container with ID starting with 893e02410559310374758e3046fa5cdb25a21a0b561b9a858a3482c6a2661e14 not found: ID does not exist"
Sep 16 18:44:34.493591 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.493569    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-909a40/distinct-2-w0-909a40-kserve-58db6d447-gznfk"]
Sep 16 18:44:34.552834 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.552800    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32" path="/var/lib/kubelet/pods/44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32/volumes"
Sep 16 18:44:34.583967 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.583938    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdb505d0-3913-4ebe-b987-b80c019a2f47-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.583967 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.583963    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.583967 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.583971    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.584148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.583979    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.584148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.583987    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdb505d0-3913-4ebe-b987-b80c019a2f47-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.584148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.583995    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l9vm\" (UniqueName: \"kubernetes.io/projected/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-kube-api-access-2l9vm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.584148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.584003    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m569\" (UniqueName: \"kubernetes.io/projected/bdb505d0-3913-4ebe-b987-b80c019a2f47-kube-api-access-9m569\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.584148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.584012    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0fe54a1-f24e-479d-b114-fb8bc7dd370f-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:34.757738 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.757641    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:34.778942 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.778908    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"]
Sep 16 18:44:34.786139 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.786109    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-909a40/premium-w0-909a40-kserve-5c49f47ffc-zk8pb"]
Sep 16 18:44:34.797044 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.797012    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"]
Sep 16 18:44:34.799733 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:34.799708    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-909a40/simulator-w0-909a40-kserve-6669d85d76-cqtqf"]
Sep 16 18:44:35.559640 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:35.559595    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:36.551441 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:36.551401    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0fe54a1-f24e-479d-b114-fb8bc7dd370f" path="/var/lib/kubelet/pods/b0fe54a1-f24e-479d-b114-fb8bc7dd370f/volumes"
Sep 16 18:44:36.551966 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:36.551951    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdb505d0-3913-4ebe-b987-b80c019a2f47" path="/var/lib/kubelet/pods/bdb505d0-3913-4ebe-b987-b80c019a2f47/volumes"
Sep 16 18:44:37.166956 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:37.166924    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:37.233078 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:37.233034    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:37.449828 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:37.449790    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:37.467371 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:37.467335    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"]
Sep 16 18:44:37.467583 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:37.467525    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27" podUID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerName="istio-proxy" containerID="cri-o://cd1429eca540283d131f6db43fc1f90bcaa48545fe66b65afc3b98ccd1038afc" gracePeriod=30
Sep 16 18:44:38.179677 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:38.179632    2562 patch_prober.go:28] interesting pod/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 18:44:38.180066 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:38.179745    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27" podUID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:44:41.477678 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.477634    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:41.565491 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.565452    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:41.827166 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.827067    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"]
Sep 16 18:44:41.827781 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.827757    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32" containerName="main"
Sep 16 18:44:41.827832 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.827786    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32" containerName="main"
Sep 16 18:44:41.827832 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.827807    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb505d0-3913-4ebe-b987-b80c019a2f47" containerName="main"
Sep 16 18:44:41.827832 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.827816    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb505d0-3913-4ebe-b987-b80c019a2f47" containerName="main"
Sep 16 18:44:41.827927 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.827841    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0fe54a1-f24e-479d-b114-fb8bc7dd370f" containerName="main"
Sep 16 18:44:41.827927 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.827850    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0fe54a1-f24e-479d-b114-fb8bc7dd370f" containerName="main"
Sep 16 18:44:41.827994 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.827983    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb505d0-3913-4ebe-b987-b80c019a2f47" containerName="main"
Sep 16 18:44:41.828025 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.827997    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0fe54a1-f24e-479d-b114-fb8bc7dd370f" containerName="main"
Sep 16 18:44:41.828025 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.828008    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ac4ea4-53cc-4fa6-a799-ad93c8ee5b32" containerName="main"
Sep 16 18:44:41.831256 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.831228    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.834291 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.834258    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8edc30\"/\"auth-test-model-8edc30-kserve-self-signed-certs\""
Sep 16 18:44:41.835855 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.835829    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8edc30\"/\"default-dockercfg-k2ls4\""
Sep 16 18:44:41.836213 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.835899    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8edc30\"/\"kube-root-ca.crt\""
Sep 16 18:44:41.836828 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.835961    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8edc30\"/\"openshift-service-ca.crt\""
Sep 16 18:44:41.840833 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.840699    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"]
Sep 16 18:44:41.861087 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.861029    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqzv5\" (UniqueName: \"kubernetes.io/projected/48453f2d-c721-4699-ab46-0c713911176d-kube-api-access-jqzv5\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.861314 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.861157    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-dshm\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.861314 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.861213    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-tmp-dir\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.861314 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.861240    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-home\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.861314 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.861273    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-model-cache\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.861538 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.861339    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48453f2d-c721-4699-ab46-0c713911176d-tls-certs\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.861647 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.861626    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:41.962352 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.962306    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqzv5\" (UniqueName: \"kubernetes.io/projected/48453f2d-c721-4699-ab46-0c713911176d-kube-api-access-jqzv5\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.962583 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.962381    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-dshm\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.962583 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.962415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-tmp-dir\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.962583 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.962431    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-home\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.962583 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.962519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-model-cache\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.962799 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.962611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48453f2d-c721-4699-ab46-0c713911176d-tls-certs\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.962965 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.962942    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-home\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.963035 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.963017    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-tmp-dir\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.963087 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.963054    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-model-cache\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.964892 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.964868    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-dshm\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.965073 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.965055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48453f2d-c721-4699-ab46-0c713911176d-tls-certs\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:41.972692 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:41.972653    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqzv5\" (UniqueName: \"kubernetes.io/projected/48453f2d-c721-4699-ab46-0c713911176d-kube-api-access-jqzv5\") pod \"auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") " pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:42.145396 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.145290    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:42.427215 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.427175    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"]
Sep 16 18:44:42.428391 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:44:42.428364    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48453f2d_c721_4699_ab46_0c713911176d.slice/crio-a11aba6c1e233681197a8b590d1189bebd7e196378de6c3e0710ecc9549d280e WatchSource:0}: Error finding container a11aba6c1e233681197a8b590d1189bebd7e196378de6c3e0710ecc9549d280e: Status 404 returned error can't find the container with id a11aba6c1e233681197a8b590d1189bebd7e196378de6c3e0710ecc9549d280e
Sep 16 18:44:42.521048 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.521011    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7" event={"ID":"48453f2d-c721-4699-ab46-0c713911176d","Type":"ContainerStarted","Data":"a11aba6c1e233681197a8b590d1189bebd7e196378de6c3e0710ecc9549d280e"}
Sep 16 18:44:42.722368 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.722340    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:44:42.771227 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.771184    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-data\") pod \"81ae555d-de1f-4a84-9788-84ffc4816fd9\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") "
Sep 16 18:44:42.771437 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.771238    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-workload-certs\") pod \"81ae555d-de1f-4a84-9788-84ffc4816fd9\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") "
Sep 16 18:44:42.771437 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.771292    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-envoy\") pod \"81ae555d-de1f-4a84-9788-84ffc4816fd9\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") "
Sep 16 18:44:42.771437 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.771315    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-workload-socket\") pod \"81ae555d-de1f-4a84-9788-84ffc4816fd9\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") "
Sep 16 18:44:42.771437 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.771340    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-token\") pod \"81ae555d-de1f-4a84-9788-84ffc4816fd9\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") "
Sep 16 18:44:42.771437 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.771433    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ae555d-de1f-4a84-9788-84ffc4816fd9-istiod-ca-cert\") pod \"81ae555d-de1f-4a84-9788-84ffc4816fd9\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") "
Sep 16 18:44:42.771742 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.771470    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-credential-socket\") pod \"81ae555d-de1f-4a84-9788-84ffc4816fd9\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") "
Sep 16 18:44:42.771742 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.771538    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbf4w\" (UniqueName: \"kubernetes.io/projected/81ae555d-de1f-4a84-9788-84ffc4816fd9-kube-api-access-bbf4w\") pod \"81ae555d-de1f-4a84-9788-84ffc4816fd9\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") "
Sep 16 18:44:42.771742 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.771579    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-data" (OuterVolumeSpecName: "istio-data") pod "81ae555d-de1f-4a84-9788-84ffc4816fd9" (UID: "81ae555d-de1f-4a84-9788-84ffc4816fd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:42.771742 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.771628    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81ae555d-de1f-4a84-9788-84ffc4816fd9" (UID: "81ae555d-de1f-4a84-9788-84ffc4816fd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:42.771742 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.771614    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-podinfo\") pod \"81ae555d-de1f-4a84-9788-84ffc4816fd9\" (UID: \"81ae555d-de1f-4a84-9788-84ffc4816fd9\") "
Sep 16 18:44:42.772007 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.771851    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81ae555d-de1f-4a84-9788-84ffc4816fd9" (UID: "81ae555d-de1f-4a84-9788-84ffc4816fd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:42.772007 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.771880    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81ae555d-de1f-4a84-9788-84ffc4816fd9" (UID: "81ae555d-de1f-4a84-9788-84ffc4816fd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:42.772007 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.771907    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ae555d-de1f-4a84-9788-84ffc4816fd9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81ae555d-de1f-4a84-9788-84ffc4816fd9" (UID: "81ae555d-de1f-4a84-9788-84ffc4816fd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:42.772158 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.772121    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-data\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:42.772158 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.772143    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-workload-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:42.772158 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.772157    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-workload-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:42.772271 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.772170    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ae555d-de1f-4a84-9788-84ffc4816fd9-istiod-ca-cert\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:42.772271 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.772184    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-credential-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:42.774359 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.774334    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81ae555d-de1f-4a84-9788-84ffc4816fd9" (UID: "81ae555d-de1f-4a84-9788-84ffc4816fd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:42.774458 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.774360    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-token" (OuterVolumeSpecName: "istio-token") pod "81ae555d-de1f-4a84-9788-84ffc4816fd9" (UID: "81ae555d-de1f-4a84-9788-84ffc4816fd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:42.774498 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.774461    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81ae555d-de1f-4a84-9788-84ffc4816fd9" (UID: "81ae555d-de1f-4a84-9788-84ffc4816fd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:44:42.774699 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.774676    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ae555d-de1f-4a84-9788-84ffc4816fd9-kube-api-access-bbf4w" (OuterVolumeSpecName: "kube-api-access-bbf4w") pod "81ae555d-de1f-4a84-9788-84ffc4816fd9" (UID: "81ae555d-de1f-4a84-9788-84ffc4816fd9"). InnerVolumeSpecName "kube-api-access-bbf4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:42.873580 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.873496    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-podinfo\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:42.873580 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.873535    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-envoy\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:42.873580 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.873585    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ae555d-de1f-4a84-9788-84ffc4816fd9-istio-token\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:42.873846 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:42.873598    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbf4w\" (UniqueName: \"kubernetes.io/projected/81ae555d-de1f-4a84-9788-84ffc4816fd9-kube-api-access-bbf4w\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:44:43.538031 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:43.537991    2562 generic.go:356] "Generic (PLEG): container finished" podID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerID="cd1429eca540283d131f6db43fc1f90bcaa48545fe66b65afc3b98ccd1038afc" exitCode=0
Sep 16 18:44:43.538506 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:43.538047    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"
Sep 16 18:44:43.538506 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:43.538066    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27" event={"ID":"81ae555d-de1f-4a84-9788-84ffc4816fd9","Type":"ContainerDied","Data":"cd1429eca540283d131f6db43fc1f90bcaa48545fe66b65afc3b98ccd1038afc"}
Sep 16 18:44:43.538506 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:43.538152    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27" event={"ID":"81ae555d-de1f-4a84-9788-84ffc4816fd9","Type":"ContainerDied","Data":"6a0e7bc76e9437206aec4f3d1be74e86aae6a63aa7685524dadb54032afdd3b2"}
Sep 16 18:44:43.538506 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:43.538185    2562 scope.go:117] "RemoveContainer" containerID="cd1429eca540283d131f6db43fc1f90bcaa48545fe66b65afc3b98ccd1038afc"
Sep 16 18:44:43.540129 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:43.540101    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7" event={"ID":"48453f2d-c721-4699-ab46-0c713911176d","Type":"ContainerStarted","Data":"c3f63aa1c602bfd463a1a14a9b4b047e9a82afaa40ca8f4951504b8e193ec141"}
Sep 16 18:44:43.549301 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:43.549271    2562 scope.go:117] "RemoveContainer" containerID="8b3410298278bdf3ecf5deb02d715dde4dd3a69993c80d3a4e2fdc7602f51575"
Sep 16 18:44:43.560257 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:43.560233    2562 scope.go:117] "RemoveContainer" containerID="cd1429eca540283d131f6db43fc1f90bcaa48545fe66b65afc3b98ccd1038afc"
Sep 16 18:44:43.560637 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:44:43.560612    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1429eca540283d131f6db43fc1f90bcaa48545fe66b65afc3b98ccd1038afc\": container with ID starting with cd1429eca540283d131f6db43fc1f90bcaa48545fe66b65afc3b98ccd1038afc not found: ID does not exist" containerID="cd1429eca540283d131f6db43fc1f90bcaa48545fe66b65afc3b98ccd1038afc"
Sep 16 18:44:43.560749 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:43.560646    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1429eca540283d131f6db43fc1f90bcaa48545fe66b65afc3b98ccd1038afc"} err="failed to get container status \"cd1429eca540283d131f6db43fc1f90bcaa48545fe66b65afc3b98ccd1038afc\": rpc error: code = NotFound desc = could not find container \"cd1429eca540283d131f6db43fc1f90bcaa48545fe66b65afc3b98ccd1038afc\": container with ID starting with cd1429eca540283d131f6db43fc1f90bcaa48545fe66b65afc3b98ccd1038afc not found: ID does not exist"
Sep 16 18:44:43.560749 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:43.560665    2562 scope.go:117] "RemoveContainer" containerID="8b3410298278bdf3ecf5deb02d715dde4dd3a69993c80d3a4e2fdc7602f51575"
Sep 16 18:44:43.560940 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:44:43.560924    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b3410298278bdf3ecf5deb02d715dde4dd3a69993c80d3a4e2fdc7602f51575\": container with ID starting with 8b3410298278bdf3ecf5deb02d715dde4dd3a69993c80d3a4e2fdc7602f51575 not found: ID does not exist" containerID="8b3410298278bdf3ecf5deb02d715dde4dd3a69993c80d3a4e2fdc7602f51575"
Sep 16 18:44:43.560999 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:43.560944    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3410298278bdf3ecf5deb02d715dde4dd3a69993c80d3a4e2fdc7602f51575"} err="failed to get container status \"8b3410298278bdf3ecf5deb02d715dde4dd3a69993c80d3a4e2fdc7602f51575\": rpc error: code = NotFound desc = could not find container \"8b3410298278bdf3ecf5deb02d715dde4dd3a69993c80d3a4e2fdc7602f51575\": container with ID starting with 8b3410298278bdf3ecf5deb02d715dde4dd3a69993c80d3a4e2fdc7602f51575 not found: ID does not exist"
Sep 16 18:44:43.570325 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:43.570266    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7" podStartSLOduration=2.57024834 podStartE2EDuration="2.57024834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:43.569494903 +0000 UTC m=+1721.434398952" watchObservedRunningTime="2026-09-16 18:44:43.57024834 +0000 UTC m=+1721.435152411"
Sep 16 18:44:43.591792 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:43.591755    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"]
Sep 16 18:44:43.598164 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:43.598128    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-909a40-openshift-default-5d6dcfd97d-7ss27"]
Sep 16 18:44:44.552703 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:44.552664    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81ae555d-de1f-4a84-9788-84ffc4816fd9" path="/var/lib/kubelet/pods/81ae555d-de1f-4a84-9788-84ffc4816fd9/volumes"
Sep 16 18:44:52.145499 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:52.145456    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:52.146078 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:52.145522    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:52.163217 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:52.163185    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:52.614464 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:52.614429    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:44:55.227354 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:55.227308    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:44:56.972370 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:56.972329    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:57.076071 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:57.076027    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:57.173839 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:44:57.173802    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:00.176329 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.176293    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826405-cm5qt"]
Sep 16 18:45:00.176804 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.176790    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerName="istio-proxy"
Sep 16 18:45:00.176804 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.176804    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerName="istio-proxy"
Sep 16 18:45:00.176881 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.176826    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerName="istio-proxy"
Sep 16 18:45:00.176881 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.176832    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerName="istio-proxy"
Sep 16 18:45:00.176948 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.176912    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerName="istio-proxy"
Sep 16 18:45:00.179909 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.179890    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-cm5qt"
Sep 16 18:45:00.185074 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.185053    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:45:00.189195 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.189179    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-cm5qt"
Sep 16 18:45:00.194931 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.194908    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d8a600-29826405-dtmpq"]
Sep 16 18:45:00.196354 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.196320    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="81ae555d-de1f-4a84-9788-84ffc4816fd9" containerName="istio-proxy"
Sep 16 18:45:00.207078 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.206682    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-520999-29826405-59lzx"]
Sep 16 18:45:00.208415 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.208385    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d8a600-29826405-dtmpq"
Sep 16 18:45:00.213817 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.213777    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ee3af0-29826405-l672n"]
Sep 16 18:45:00.214045 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.213866    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-520999-29826405-59lzx"
Sep 16 18:45:00.218033 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.218002    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826405-cm5qt"]
Sep 16 18:45:00.218174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.218106    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ee3af0-29826405-l672n"
Sep 16 18:45:00.229370 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.229318    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab83f76b-29826405-dr5lp"]
Sep 16 18:45:00.236774 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.236743    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d8a600-29826405-dtmpq"]
Sep 16 18:45:00.237224 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.237025    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab83f76b-29826405-dr5lp"
Sep 16 18:45:00.240608 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.240300    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ee3af0-29826405-l672n"]
Sep 16 18:45:00.247179 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.247022    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d8a600-29826405-dtmpq"
Sep 16 18:45:00.249403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.249375    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab83f76b-29826405-dr5lp"]
Sep 16 18:45:00.263557 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.263257    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-520999-29826405-59lzx"]
Sep 16 18:45:00.264090 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.263884    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-520999-29826405-59lzx"
Sep 16 18:45:00.318150 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.318118    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ee3af0-29826405-l672n"
Sep 16 18:45:00.330339 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.330255    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab83f76b-29826405-dr5lp"
Sep 16 18:45:00.504162 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:45:00.504120    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbece99ba_a6cb_424e_9187_67fd8e616034.slice/crio-6ce55acbe0478cf817a85d4caaa4b399ac28c81ec64abee9ae066eef7ec55c57 WatchSource:0}: Error finding container 6ce55acbe0478cf817a85d4caaa4b399ac28c81ec64abee9ae066eef7ec55c57: Status 404 returned error can't find the container with id 6ce55acbe0478cf817a85d4caaa4b399ac28c81ec64abee9ae066eef7ec55c57
Sep 16 18:45:00.504989 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.504905    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826405-cm5qt"]
Sep 16 18:45:00.569244 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.569220    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d8a600-29826405-dtmpq"]
Sep 16 18:45:00.570994 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:45:00.570966    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d98ec06_4e04_4e74_ab0b_fe753b16ac64.slice/crio-7f224da8fd8ecb9f4bbce9a105c37479032e5dcce3d70935dfadde5c4c5a9d25 WatchSource:0}: Error finding container 7f224da8fd8ecb9f4bbce9a105c37479032e5dcce3d70935dfadde5c4c5a9d25: Status 404 returned error can't find the container with id 7f224da8fd8ecb9f4bbce9a105c37479032e5dcce3d70935dfadde5c4c5a9d25
Sep 16 18:45:00.595410 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.595384    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-520999-29826405-59lzx"]
Sep 16 18:45:00.602832 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:45:00.602803    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d858b9e_8c3f_4d62_b772_a0e525e34117.slice/crio-ae50bcbbda3379b8f16e70ffd5cb7f9dd7ab7bbca6f502573abe183d2ab493f4 WatchSource:0}: Error finding container ae50bcbbda3379b8f16e70ffd5cb7f9dd7ab7bbca6f502573abe183d2ab493f4: Status 404 returned error can't find the container with id ae50bcbbda3379b8f16e70ffd5cb7f9dd7ab7bbca6f502573abe183d2ab493f4
Sep 16 18:45:00.621448 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.621419    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ee3af0-29826405-l672n"]
Sep 16 18:45:00.622417 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:45:00.622391    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0c0806_c126_4ba0_a1af_f24878c45924.slice/crio-111f3b0105f46f656e357bfe778dc2f9be9b74b0e702d7fe4afedb596bcb23a6 WatchSource:0}: Error finding container 111f3b0105f46f656e357bfe778dc2f9be9b74b0e702d7fe4afedb596bcb23a6: Status 404 returned error can't find the container with id 111f3b0105f46f656e357bfe778dc2f9be9b74b0e702d7fe4afedb596bcb23a6
Sep 16 18:45:00.657729 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.657696    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab83f76b-29826405-dr5lp"]
Sep 16 18:45:00.658668 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.658639    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-520999-29826405-59lzx" event={"ID":"3d858b9e-8c3f-4d62-b772-a0e525e34117","Type":"ContainerStarted","Data":"ae50bcbbda3379b8f16e70ffd5cb7f9dd7ab7bbca6f502573abe183d2ab493f4"}
Sep 16 18:45:00.659263 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:45:00.659237    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8706b0a0_a218_4870_b624_442d2c21db47.slice/crio-17dc76a63f6e13a8307c14e2564347057d6e8a0b27d56f4b4b7620c388c18f28 WatchSource:0}: Error finding container 17dc76a63f6e13a8307c14e2564347057d6e8a0b27d56f4b4b7620c388c18f28: Status 404 returned error can't find the container with id 17dc76a63f6e13a8307c14e2564347057d6e8a0b27d56f4b4b7620c388c18f28
Sep 16 18:45:00.660177 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.660153    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-cm5qt" event={"ID":"bece99ba-a6cb-424e-9187-67fd8e616034","Type":"ContainerStarted","Data":"6ce55acbe0478cf817a85d4caaa4b399ac28c81ec64abee9ae066eef7ec55c57"}
Sep 16 18:45:00.665177 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.665143    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ee3af0-29826405-l672n" event={"ID":"7d0c0806-c126-4ba0-a1af-f24878c45924","Type":"ContainerStarted","Data":"111f3b0105f46f656e357bfe778dc2f9be9b74b0e702d7fe4afedb596bcb23a6"}
Sep 16 18:45:00.666687 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:00.666663    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d8a600-29826405-dtmpq" event={"ID":"6d98ec06-4e04-4e74-ab0b-fe753b16ac64","Type":"ContainerStarted","Data":"7f224da8fd8ecb9f4bbce9a105c37479032e5dcce3d70935dfadde5c4c5a9d25"}
Sep 16 18:45:01.674995 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:01.674956    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab83f76b-29826405-dr5lp" event={"ID":"8706b0a0-a218-4870-b624-442d2c21db47","Type":"ContainerStarted","Data":"9118e0705004d7d4878eb4964f5e33b5773202b5aef95830907bca6f47598fcd"}
Sep 16 18:45:01.675428 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:01.675016    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab83f76b-29826405-dr5lp" event={"ID":"8706b0a0-a218-4870-b624-442d2c21db47","Type":"ContainerStarted","Data":"17dc76a63f6e13a8307c14e2564347057d6e8a0b27d56f4b4b7620c388c18f28"}
Sep 16 18:45:01.676669 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:01.676643    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ee3af0-29826405-l672n" event={"ID":"7d0c0806-c126-4ba0-a1af-f24878c45924","Type":"ContainerStarted","Data":"8d6a926f569a7d58a87bc2f4a86f070052f9ce4909d867fa05657c8b37fb52dd"}
Sep 16 18:45:01.678198 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:01.678173    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d8a600-29826405-dtmpq" event={"ID":"6d98ec06-4e04-4e74-ab0b-fe753b16ac64","Type":"ContainerStarted","Data":"9767cf859b1fc0adef47c8485e41260af217d61596713f30812969cad3978fa9"}
Sep 16 18:45:01.679789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:01.679767    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-520999-29826405-59lzx" event={"ID":"3d858b9e-8c3f-4d62-b772-a0e525e34117","Type":"ContainerStarted","Data":"a1096d1b9b4bae09d70ebb7d0a4f53cf9e124862bc1b432978155d22d62a0461"}
Sep 16 18:45:01.681389 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:01.681366    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-cm5qt" event={"ID":"bece99ba-a6cb-424e-9187-67fd8e616034","Type":"ContainerStarted","Data":"fece6ba20aa219c6653edcf43ca00bb8c8d6c7e5f211182c181fff256ac324be"}
Sep 16 18:45:01.694427 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:01.694381    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab83f76b-29826405-dr5lp" podStartSLOduration=1.694365396 podStartE2EDuration="1.694365396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:01.692848188 +0000 UTC m=+1739.557752227" watchObservedRunningTime="2026-09-16 18:45:01.694365396 +0000 UTC m=+1739.559269444"
Sep 16 18:45:01.710679 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:01.710625    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-520999-29826405-59lzx" podStartSLOduration=1.710609437 podStartE2EDuration="1.710609437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:01.709593649 +0000 UTC m=+1739.574497700" watchObservedRunningTime="2026-09-16 18:45:01.710609437 +0000 UTC m=+1739.575513485"
Sep 16 18:45:01.728225 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:01.728179    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ee3af0-29826405-l672n" podStartSLOduration=1.728165008 podStartE2EDuration="1.728165008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:01.726490075 +0000 UTC m=+1739.591394123" watchObservedRunningTime="2026-09-16 18:45:01.728165008 +0000 UTC m=+1739.593069056"
Sep 16 18:45:01.748405 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:01.748350    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d8a600-29826405-dtmpq" podStartSLOduration=1.748333769 podStartE2EDuration="1.748333769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:01.745450488 +0000 UTC m=+1739.610354531" watchObservedRunningTime="2026-09-16 18:45:01.748333769 +0000 UTC m=+1739.613237817"
Sep 16 18:45:01.772945 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:01.772888    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-cm5qt" podStartSLOduration=1.772870534 podStartE2EDuration="1.772870534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:01.769628441 +0000 UTC m=+1739.634532483" watchObservedRunningTime="2026-09-16 18:45:01.772870534 +0000 UTC m=+1739.637774582"
Sep 16 18:45:02.689526 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:02.689495    2562 generic.go:356] "Generic (PLEG): container finished" podID="3d858b9e-8c3f-4d62-b772-a0e525e34117" containerID="a1096d1b9b4bae09d70ebb7d0a4f53cf9e124862bc1b432978155d22d62a0461" exitCode=0
Sep 16 18:45:02.689964 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:02.689583    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-520999-29826405-59lzx" event={"ID":"3d858b9e-8c3f-4d62-b772-a0e525e34117","Type":"ContainerDied","Data":"a1096d1b9b4bae09d70ebb7d0a4f53cf9e124862bc1b432978155d22d62a0461"}
Sep 16 18:45:02.691208 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:02.691185    2562 generic.go:356] "Generic (PLEG): container finished" podID="bece99ba-a6cb-424e-9187-67fd8e616034" containerID="fece6ba20aa219c6653edcf43ca00bb8c8d6c7e5f211182c181fff256ac324be" exitCode=0
Sep 16 18:45:02.691333 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:02.691268    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-cm5qt" event={"ID":"bece99ba-a6cb-424e-9187-67fd8e616034","Type":"ContainerDied","Data":"fece6ba20aa219c6653edcf43ca00bb8c8d6c7e5f211182c181fff256ac324be"}
Sep 16 18:45:02.692824 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:02.692801    2562 generic.go:356] "Generic (PLEG): container finished" podID="8706b0a0-a218-4870-b624-442d2c21db47" containerID="9118e0705004d7d4878eb4964f5e33b5773202b5aef95830907bca6f47598fcd" exitCode=0
Sep 16 18:45:02.692959 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:02.692878    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab83f76b-29826405-dr5lp" event={"ID":"8706b0a0-a218-4870-b624-442d2c21db47","Type":"ContainerDied","Data":"9118e0705004d7d4878eb4964f5e33b5773202b5aef95830907bca6f47598fcd"}
Sep 16 18:45:02.694576 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:02.694558    2562 generic.go:356] "Generic (PLEG): container finished" podID="7d0c0806-c126-4ba0-a1af-f24878c45924" containerID="8d6a926f569a7d58a87bc2f4a86f070052f9ce4909d867fa05657c8b37fb52dd" exitCode=0
Sep 16 18:45:02.694662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:02.694640    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ee3af0-29826405-l672n" event={"ID":"7d0c0806-c126-4ba0-a1af-f24878c45924","Type":"ContainerDied","Data":"8d6a926f569a7d58a87bc2f4a86f070052f9ce4909d867fa05657c8b37fb52dd"}
Sep 16 18:45:02.696170 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:02.696151    2562 generic.go:356] "Generic (PLEG): container finished" podID="6d98ec06-4e04-4e74-ab0b-fe753b16ac64" containerID="9767cf859b1fc0adef47c8485e41260af217d61596713f30812969cad3978fa9" exitCode=0
Sep 16 18:45:02.696249 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:02.696175    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d8a600-29826405-dtmpq" event={"ID":"6d98ec06-4e04-4e74-ab0b-fe753b16ac64","Type":"ContainerDied","Data":"9767cf859b1fc0adef47c8485e41260af217d61596713f30812969cad3978fa9"}
Sep 16 18:45:03.892343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:03.892305    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ee3af0-29826405-l672n"
Sep 16 18:45:03.964788 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:03.964718    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab83f76b-29826405-dr5lp"
Sep 16 18:45:03.967652 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:03.967625    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-cm5qt"
Sep 16 18:45:03.973941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:03.973918    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d8a600-29826405-dtmpq"
Sep 16 18:45:03.977844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:03.977817    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-520999-29826405-59lzx"
Sep 16 18:45:04.715514 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:04.715475    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-520999-29826405-59lzx" event={"ID":"3d858b9e-8c3f-4d62-b772-a0e525e34117","Type":"ContainerDied","Data":"ae50bcbbda3379b8f16e70ffd5cb7f9dd7ab7bbca6f502573abe183d2ab493f4"}
Sep 16 18:45:04.715514 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:04.715500    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-520999-29826405-59lzx"
Sep 16 18:45:04.715789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:04.715567    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae50bcbbda3379b8f16e70ffd5cb7f9dd7ab7bbca6f502573abe183d2ab493f4"
Sep 16 18:45:04.716988 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:04.716966    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-cm5qt"
Sep 16 18:45:04.717125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:04.716998    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826405-cm5qt" event={"ID":"bece99ba-a6cb-424e-9187-67fd8e616034","Type":"ContainerDied","Data":"6ce55acbe0478cf817a85d4caaa4b399ac28c81ec64abee9ae066eef7ec55c57"}
Sep 16 18:45:04.717125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:04.717049    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ce55acbe0478cf817a85d4caaa4b399ac28c81ec64abee9ae066eef7ec55c57"
Sep 16 18:45:04.718691 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:04.718666    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab83f76b-29826405-dr5lp" event={"ID":"8706b0a0-a218-4870-b624-442d2c21db47","Type":"ContainerDied","Data":"17dc76a63f6e13a8307c14e2564347057d6e8a0b27d56f4b4b7620c388c18f28"}
Sep 16 18:45:04.718816 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:04.718700    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab83f76b-29826405-dr5lp"
Sep 16 18:45:04.718816 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:04.718711    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17dc76a63f6e13a8307c14e2564347057d6e8a0b27d56f4b4b7620c388c18f28"
Sep 16 18:45:04.720372 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:04.720350    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ee3af0-29826405-l672n"
Sep 16 18:45:04.720500 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:04.720350    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ee3af0-29826405-l672n" event={"ID":"7d0c0806-c126-4ba0-a1af-f24878c45924","Type":"ContainerDied","Data":"111f3b0105f46f656e357bfe778dc2f9be9b74b0e702d7fe4afedb596bcb23a6"}
Sep 16 18:45:04.720593 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:04.720502    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="111f3b0105f46f656e357bfe778dc2f9be9b74b0e702d7fe4afedb596bcb23a6"
Sep 16 18:45:04.722085 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:04.722060    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d8a600-29826405-dtmpq" event={"ID":"6d98ec06-4e04-4e74-ab0b-fe753b16ac64","Type":"ContainerDied","Data":"7f224da8fd8ecb9f4bbce9a105c37479032e5dcce3d70935dfadde5c4c5a9d25"}
Sep 16 18:45:04.722085 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:04.722080    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d8a600-29826405-dtmpq"
Sep 16 18:45:04.722260 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:04.722104    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f224da8fd8ecb9f4bbce9a105c37479032e5dcce3d70935dfadde5c4c5a9d25"
Sep 16 18:45:06.753793 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:06.753760    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:10.741088 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:10.741042    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"]
Sep 16 18:45:10.764266 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:10.764225    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:22.312308 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:22.312265    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:45:25.762284 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:25.762218    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb" podUID="aeb69706-c123-4d2f-ac05-fa3438b2952e" containerName="main" containerID="cri-o://9bb0ee65e02ffddf57cc6a5374109d83b088029e21ec483a2c1ee9f6af4f16ee" gracePeriod=45
Sep 16 18:45:26.026073 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.026004    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:45:26.123987 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.123953    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-home\") pod \"aeb69706-c123-4d2f-ac05-fa3438b2952e\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") "
Sep 16 18:45:26.123987 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.123989    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeb69706-c123-4d2f-ac05-fa3438b2952e-tls-certs\") pod \"aeb69706-c123-4d2f-ac05-fa3438b2952e\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") "
Sep 16 18:45:26.124190 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.124056    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-dshm\") pod \"aeb69706-c123-4d2f-ac05-fa3438b2952e\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") "
Sep 16 18:45:26.124190 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.124080    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-model-cache\") pod \"aeb69706-c123-4d2f-ac05-fa3438b2952e\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") "
Sep 16 18:45:26.124190 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.124123    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4kjl\" (UniqueName: \"kubernetes.io/projected/aeb69706-c123-4d2f-ac05-fa3438b2952e-kube-api-access-q4kjl\") pod \"aeb69706-c123-4d2f-ac05-fa3438b2952e\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") "
Sep 16 18:45:26.124190 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.124159    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-tmp-dir\") pod \"aeb69706-c123-4d2f-ac05-fa3438b2952e\" (UID: \"aeb69706-c123-4d2f-ac05-fa3438b2952e\") "
Sep 16 18:45:26.124383 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.124222    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-home" (OuterVolumeSpecName: "home") pod "aeb69706-c123-4d2f-ac05-fa3438b2952e" (UID: "aeb69706-c123-4d2f-ac05-fa3438b2952e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:26.124450 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.124404    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-model-cache" (OuterVolumeSpecName: "model-cache") pod "aeb69706-c123-4d2f-ac05-fa3438b2952e" (UID: "aeb69706-c123-4d2f-ac05-fa3438b2952e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:26.124510 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.124476    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aeb69706-c123-4d2f-ac05-fa3438b2952e" (UID: "aeb69706-c123-4d2f-ac05-fa3438b2952e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:26.124654 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.124634    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:45:26.124719 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.124657    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:45:26.124719 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.124671    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:45:26.126368 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.126342    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-dshm" (OuterVolumeSpecName: "dshm") pod "aeb69706-c123-4d2f-ac05-fa3438b2952e" (UID: "aeb69706-c123-4d2f-ac05-fa3438b2952e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:26.126496 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.126430    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb69706-c123-4d2f-ac05-fa3438b2952e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aeb69706-c123-4d2f-ac05-fa3438b2952e" (UID: "aeb69706-c123-4d2f-ac05-fa3438b2952e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:26.126574 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.126528    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb69706-c123-4d2f-ac05-fa3438b2952e-kube-api-access-q4kjl" (OuterVolumeSpecName: "kube-api-access-q4kjl") pod "aeb69706-c123-4d2f-ac05-fa3438b2952e" (UID: "aeb69706-c123-4d2f-ac05-fa3438b2952e"). InnerVolumeSpecName "kube-api-access-q4kjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:26.225419 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.225385    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeb69706-c123-4d2f-ac05-fa3438b2952e-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:45:26.225419 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.225413    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeb69706-c123-4d2f-ac05-fa3438b2952e-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:45:26.225419 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.225423    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4kjl\" (UniqueName: \"kubernetes.io/projected/aeb69706-c123-4d2f-ac05-fa3438b2952e-kube-api-access-q4kjl\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:45:26.875884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.875782    2562 generic.go:356] "Generic (PLEG): container finished" podID="aeb69706-c123-4d2f-ac05-fa3438b2952e" containerID="9bb0ee65e02ffddf57cc6a5374109d83b088029e21ec483a2c1ee9f6af4f16ee" exitCode=0
Sep 16 18:45:26.875884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.875821    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"
Sep 16 18:45:26.875884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.875867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb" event={"ID":"aeb69706-c123-4d2f-ac05-fa3438b2952e","Type":"ContainerDied","Data":"9bb0ee65e02ffddf57cc6a5374109d83b088029e21ec483a2c1ee9f6af4f16ee"}
Sep 16 18:45:26.876439 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.875908    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb" event={"ID":"aeb69706-c123-4d2f-ac05-fa3438b2952e","Type":"ContainerDied","Data":"c05baf877907cace277b86df8b0c07344dc795c9e72a4d0c30b12e01a9d1d867"}
Sep 16 18:45:26.876439 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.875927    2562 scope.go:117] "RemoveContainer" containerID="9bb0ee65e02ffddf57cc6a5374109d83b088029e21ec483a2c1ee9f6af4f16ee"
Sep 16 18:45:26.888371 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.888333    2562 scope.go:117] "RemoveContainer" containerID="9bb0ee65e02ffddf57cc6a5374109d83b088029e21ec483a2c1ee9f6af4f16ee"
Sep 16 18:45:26.888859 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:45:26.888833    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb0ee65e02ffddf57cc6a5374109d83b088029e21ec483a2c1ee9f6af4f16ee\": container with ID starting with 9bb0ee65e02ffddf57cc6a5374109d83b088029e21ec483a2c1ee9f6af4f16ee not found: ID does not exist" containerID="9bb0ee65e02ffddf57cc6a5374109d83b088029e21ec483a2c1ee9f6af4f16ee"
Sep 16 18:45:26.888941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.888867    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb0ee65e02ffddf57cc6a5374109d83b088029e21ec483a2c1ee9f6af4f16ee"} err="failed to get container status \"9bb0ee65e02ffddf57cc6a5374109d83b088029e21ec483a2c1ee9f6af4f16ee\": rpc error: code = NotFound desc = could not find container \"9bb0ee65e02ffddf57cc6a5374109d83b088029e21ec483a2c1ee9f6af4f16ee\": container with ID starting with 9bb0ee65e02ffddf57cc6a5374109d83b088029e21ec483a2c1ee9f6af4f16ee not found: ID does not exist"
Sep 16 18:45:26.897124 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.897099    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"]
Sep 16 18:45:26.903779 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:26.903756    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/e2e-conflict-b-8b493098-kserve-d84fdfdc8-vfxrb"]
Sep 16 18:45:28.551202 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:28.551170    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeb69706-c123-4d2f-ac05-fa3438b2952e" path="/var/lib/kubelet/pods/aeb69706-c123-4d2f-ac05-fa3438b2952e/volumes"
Sep 16 18:45:31.502904 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.502870    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp"]
Sep 16 18:45:31.503294 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503266    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d0c0806-c126-4ba0-a1af-f24878c45924" containerName="cleanup"
Sep 16 18:45:31.503294 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503278    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d0c0806-c126-4ba0-a1af-f24878c45924" containerName="cleanup"
Sep 16 18:45:31.503294 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503286    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8706b0a0-a218-4870-b624-442d2c21db47" containerName="cleanup"
Sep 16 18:45:31.503294 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503292    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8706b0a0-a218-4870-b624-442d2c21db47" containerName="cleanup"
Sep 16 18:45:31.503423 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503303    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb69706-c123-4d2f-ac05-fa3438b2952e" containerName="main"
Sep 16 18:45:31.503423 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503309    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb69706-c123-4d2f-ac05-fa3438b2952e" containerName="main"
Sep 16 18:45:31.503423 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503318    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d858b9e-8c3f-4d62-b772-a0e525e34117" containerName="cleanup"
Sep 16 18:45:31.503423 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503324    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d858b9e-8c3f-4d62-b772-a0e525e34117" containerName="cleanup"
Sep 16 18:45:31.503423 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503342    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bece99ba-a6cb-424e-9187-67fd8e616034" containerName="cleanup"
Sep 16 18:45:31.503423 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503347    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bece99ba-a6cb-424e-9187-67fd8e616034" containerName="cleanup"
Sep 16 18:45:31.503423 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503357    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d98ec06-4e04-4e74-ab0b-fe753b16ac64" containerName="cleanup"
Sep 16 18:45:31.503423 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503362    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d98ec06-4e04-4e74-ab0b-fe753b16ac64" containerName="cleanup"
Sep 16 18:45:31.503705 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503436    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d0c0806-c126-4ba0-a1af-f24878c45924" containerName="cleanup"
Sep 16 18:45:31.503705 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503447    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8706b0a0-a218-4870-b624-442d2c21db47" containerName="cleanup"
Sep 16 18:45:31.503705 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503453    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d98ec06-4e04-4e74-ab0b-fe753b16ac64" containerName="cleanup"
Sep 16 18:45:31.503705 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503459    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d858b9e-8c3f-4d62-b772-a0e525e34117" containerName="cleanup"
Sep 16 18:45:31.503705 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503464    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bece99ba-a6cb-424e-9187-67fd8e616034" containerName="cleanup"
Sep 16 18:45:31.503705 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.503471    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb69706-c123-4d2f-ac05-fa3438b2952e" containerName="main"
Sep 16 18:45:31.508061 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.508043    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp"
Sep 16 18:45:31.511685 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.511625    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:45:31.512477 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.512441    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:45:31.515829 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.515803    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp"]
Sep 16 18:45:31.575921 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.575875    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebbcdc5-96fe-46a8-9f8d-391b611c3c72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp\" (UID: \"cebbcdc5-96fe-46a8-9f8d-391b611c3c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp"
Sep 16 18:45:31.676751 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.676713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebbcdc5-96fe-46a8-9f8d-391b611c3c72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp\" (UID: \"cebbcdc5-96fe-46a8-9f8d-391b611c3c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp"
Sep 16 18:45:31.677558 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.677517    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebbcdc5-96fe-46a8-9f8d-391b611c3c72-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp\" (UID: \"cebbcdc5-96fe-46a8-9f8d-391b611c3c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp"
Sep 16 18:45:31.821988 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.821905    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp"
Sep 16 18:45:31.832427 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:31.832401    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:32.106620 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:32.106595    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp"]
Sep 16 18:45:32.107807 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:45:32.107777    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebbcdc5_96fe_46a8_9f8d_391b611c3c72.slice/crio-7c08072522cb7490fa765cb63f77820736701b634b2dd42970c380501e0bff1a WatchSource:0}: Error finding container 7c08072522cb7490fa765cb63f77820736701b634b2dd42970c380501e0bff1a: Status 404 returned error can't find the container with id 7c08072522cb7490fa765cb63f77820736701b634b2dd42970c380501e0bff1a
Sep 16 18:45:32.759417 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:32.759369    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:32.924303 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:32.924263    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp" event={"ID":"cebbcdc5-96fe-46a8-9f8d-391b611c3c72","Type":"ContainerStarted","Data":"91da309506063b32dfd6f75f307c9787892cb6f0d91b6e5d6357760404207404"}
Sep 16 18:45:32.924508 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:32.924325    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp" event={"ID":"cebbcdc5-96fe-46a8-9f8d-391b611c3c72","Type":"ContainerStarted","Data":"7c08072522cb7490fa765cb63f77820736701b634b2dd42970c380501e0bff1a"}
Sep 16 18:45:32.945478 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:32.945422    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp" podStartSLOduration=1.94540483 podStartE2EDuration="1.94540483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:32.944641423 +0000 UTC m=+1770.809545482" watchObservedRunningTime="2026-09-16 18:45:32.94540483 +0000 UTC m=+1770.810308872"
Sep 16 18:45:33.473249 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:33.473211    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:33.931758 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:33.931662    2562 generic.go:356] "Generic (PLEG): container finished" podID="cebbcdc5-96fe-46a8-9f8d-391b611c3c72" containerID="91da309506063b32dfd6f75f307c9787892cb6f0d91b6e5d6357760404207404" exitCode=0
Sep 16 18:45:33.932142 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:33.931748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp" event={"ID":"cebbcdc5-96fe-46a8-9f8d-391b611c3c72","Type":"ContainerDied","Data":"91da309506063b32dfd6f75f307c9787892cb6f0d91b6e5d6357760404207404"}
Sep 16 18:45:34.194207 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:34.194120    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:35.082168 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:35.082142    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp"
Sep 16 18:45:35.125487 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:35.125451    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:35.224944 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:35.224913    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebbcdc5-96fe-46a8-9f8d-391b611c3c72-maas-api-service-ca\") pod \"cebbcdc5-96fe-46a8-9f8d-391b611c3c72\" (UID: \"cebbcdc5-96fe-46a8-9f8d-391b611c3c72\") "
Sep 16 18:45:35.225315 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:35.225287    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cebbcdc5-96fe-46a8-9f8d-391b611c3c72-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cebbcdc5-96fe-46a8-9f8d-391b611c3c72" (UID: "cebbcdc5-96fe-46a8-9f8d-391b611c3c72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:35.326587 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:35.326518    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebbcdc5-96fe-46a8-9f8d-391b611c3c72-maas-api-service-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:45:35.946598 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:35.946564    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp" event={"ID":"cebbcdc5-96fe-46a8-9f8d-391b611c3c72","Type":"ContainerDied","Data":"7c08072522cb7490fa765cb63f77820736701b634b2dd42970c380501e0bff1a"}
Sep 16 18:45:35.946775 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:35.946620    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c08072522cb7490fa765cb63f77820736701b634b2dd42970c380501e0bff1a"
Sep 16 18:45:35.946775 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:35.946571    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp"
Sep 16 18:45:35.976756 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:35.976682    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:36.723051 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:36.723013    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:38.056859 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:38.056826    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:39.103296 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:39.103259    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:41.886515 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:41.886481    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr"]
Sep 16 18:45:41.886912 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:41.886898    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cebbcdc5-96fe-46a8-9f8d-391b611c3c72" containerName="revoke-keys"
Sep 16 18:45:41.886912 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:41.886914    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cebbcdc5-96fe-46a8-9f8d-391b611c3c72" containerName="revoke-keys"
Sep 16 18:45:41.887003 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:41.886993    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cebbcdc5-96fe-46a8-9f8d-391b611c3c72" containerName="revoke-keys"
Sep 16 18:45:41.890029 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:41.890011    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr"
Sep 16 18:45:41.898225 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:41.898203    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:45:41.898300 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:41.898214    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:45:41.913748 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:41.913717    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr"]
Sep 16 18:45:42.002063 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:42.002005    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b55ad86-13ad-427a-8681-f2e5032cb22c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr\" (UID: \"1b55ad86-13ad-427a-8681-f2e5032cb22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr"
Sep 16 18:45:42.103068 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:42.103028    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b55ad86-13ad-427a-8681-f2e5032cb22c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr\" (UID: \"1b55ad86-13ad-427a-8681-f2e5032cb22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr"
Sep 16 18:45:42.103723 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:42.103702    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b55ad86-13ad-427a-8681-f2e5032cb22c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr\" (UID: \"1b55ad86-13ad-427a-8681-f2e5032cb22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr"
Sep 16 18:45:42.200359 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:42.200334    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr"
Sep 16 18:45:42.405778 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:42.405737    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:42.481429 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:42.481402    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr"]
Sep 16 18:45:42.483490 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:45:42.483460    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b55ad86_13ad_427a_8681_f2e5032cb22c.slice/crio-74e0116852aaed61f94ac4614290dd5dfaf23a42aa35f4e49f45f8e414699a79 WatchSource:0}: Error finding container 74e0116852aaed61f94ac4614290dd5dfaf23a42aa35f4e49f45f8e414699a79: Status 404 returned error can't find the container with id 74e0116852aaed61f94ac4614290dd5dfaf23a42aa35f4e49f45f8e414699a79
Sep 16 18:45:42.993326 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:42.993285    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr" event={"ID":"1b55ad86-13ad-427a-8681-f2e5032cb22c","Type":"ContainerStarted","Data":"cd306b387da7cc7a64ecc1b43335538d21ada56a10044421582f75cbf3bf6a18"}
Sep 16 18:45:42.993761 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:42.993341    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr" event={"ID":"1b55ad86-13ad-427a-8681-f2e5032cb22c","Type":"ContainerStarted","Data":"74e0116852aaed61f94ac4614290dd5dfaf23a42aa35f4e49f45f8e414699a79"}
Sep 16 18:45:43.012569 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:43.012501    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr" podStartSLOduration=2.012486166 podStartE2EDuration="2.012486166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:43.011515603 +0000 UTC m=+1780.876419650" watchObservedRunningTime="2026-09-16 18:45:43.012486166 +0000 UTC m=+1780.877390215"
Sep 16 18:45:43.321030 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:43.320985    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:44.001708 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:44.001674    2562 generic.go:356] "Generic (PLEG): container finished" podID="1b55ad86-13ad-427a-8681-f2e5032cb22c" containerID="cd306b387da7cc7a64ecc1b43335538d21ada56a10044421582f75cbf3bf6a18" exitCode=0
Sep 16 18:45:44.002056 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:44.001760    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr" event={"ID":"1b55ad86-13ad-427a-8681-f2e5032cb22c","Type":"ContainerDied","Data":"cd306b387da7cc7a64ecc1b43335538d21ada56a10044421582f75cbf3bf6a18"}
Sep 16 18:45:44.227771 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:44.227735    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:45.083333 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:45.083298    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:45.154959 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:45.154929    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr"
Sep 16 18:45:45.335355 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:45.335263    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b55ad86-13ad-427a-8681-f2e5032cb22c-maas-api-service-ca\") pod \"1b55ad86-13ad-427a-8681-f2e5032cb22c\" (UID: \"1b55ad86-13ad-427a-8681-f2e5032cb22c\") "
Sep 16 18:45:45.335730 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:45.335698    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b55ad86-13ad-427a-8681-f2e5032cb22c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b55ad86-13ad-427a-8681-f2e5032cb22c" (UID: "1b55ad86-13ad-427a-8681-f2e5032cb22c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:45.436042 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:45.436007    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b55ad86-13ad-427a-8681-f2e5032cb22c-maas-api-service-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:45:45.957824 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:45.957775    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:46.020039 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:46.020009    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr"
Sep 16 18:45:46.020215 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:46.020008    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr" event={"ID":"1b55ad86-13ad-427a-8681-f2e5032cb22c","Type":"ContainerDied","Data":"74e0116852aaed61f94ac4614290dd5dfaf23a42aa35f4e49f45f8e414699a79"}
Sep 16 18:45:46.020215 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:46.020138    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74e0116852aaed61f94ac4614290dd5dfaf23a42aa35f4e49f45f8e414699a79"
Sep 16 18:45:46.898724 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:46.898687    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:47.695421 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:47.695388    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab83f76b-29826405-dr5lp"]
Sep 16 18:45:47.698239 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:47.698212    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:47.701995 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:47.701967    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab83f76b-29826405-dr5lp"]
Sep 16 18:45:47.764555 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:47.764498    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:47.865144 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:47.865100    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:47.936255 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:47.936194    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp"]
Sep 16 18:45:47.940099 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:47.940073    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab83f76b-67mwp"]
Sep 16 18:45:47.956722 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:47.956689    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:48.031592 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:48.031534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:48.295615 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:48.295503    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:48.509383 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:48.509351    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:48.550718 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:48.550626    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8706b0a0-a218-4870-b624-442d2c21db47" path="/var/lib/kubelet/pods/8706b0a0-a218-4870-b624-442d2c21db47/volumes"
Sep 16 18:45:48.551077 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:48.551055    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cebbcdc5-96fe-46a8-9f8d-391b611c3c72" path="/var/lib/kubelet/pods/cebbcdc5-96fe-46a8-9f8d-391b611c3c72/volumes"
Sep 16 18:45:49.457073 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:49.457042    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:50.585834 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:50.585796    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:51.335024 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.334977    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"]
Sep 16 18:45:51.335444 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.335429    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b55ad86-13ad-427a-8681-f2e5032cb22c" containerName="revoke-keys"
Sep 16 18:45:51.335490 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.335445    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b55ad86-13ad-427a-8681-f2e5032cb22c" containerName="revoke-keys"
Sep 16 18:45:51.335573 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.335538    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b55ad86-13ad-427a-8681-f2e5032cb22c" containerName="revoke-keys"
Sep 16 18:45:51.338855 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.338837    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.342007 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.341979    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d8a600\"/\"distinct-w4-d8a600-kserve-self-signed-certs\""
Sep 16 18:45:51.372979 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.372942    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"]
Sep 16 18:45:51.389813 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.389774    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-home\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.389966 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.389882    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8rxn\" (UniqueName: \"kubernetes.io/projected/60248414-4995-4f51-b233-5dd9f38968a8-kube-api-access-d8rxn\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.389966 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.389918    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-dshm\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.389966 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.389938    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-model-cache\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.389966 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.389953    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-tmp-dir\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.390143 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.390015    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60248414-4995-4f51-b233-5dd9f38968a8-tls-certs\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.430094 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.430056    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:51.491040 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.491002    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-home\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.491246 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.491075    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8rxn\" (UniqueName: \"kubernetes.io/projected/60248414-4995-4f51-b233-5dd9f38968a8-kube-api-access-d8rxn\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.491246 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.491100    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-dshm\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.491353 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.491242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-model-cache\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.491353 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.491283    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-tmp-dir\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.491353 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.491332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60248414-4995-4f51-b233-5dd9f38968a8-tls-certs\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.491529 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.491481    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-home\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.491651 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.491628    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-model-cache\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.491739 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.491652    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-tmp-dir\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.493417 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.493396    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-dshm\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.493783 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.493762    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60248414-4995-4f51-b233-5dd9f38968a8-tls-certs\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.501005 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.500980    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8rxn\" (UniqueName: \"kubernetes.io/projected/60248414-4995-4f51-b233-5dd9f38968a8-kube-api-access-d8rxn\") pod \"distinct-w4-d8a600-kserve-5684b96bc6-dc76d\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.614160 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.614071    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:51.648253 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.648217    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:45:51.946013 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:51.945978    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"]
Sep 16 18:45:51.947151 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:45:51.947125    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60248414_4995_4f51_b233_5dd9f38968a8.slice/crio-76677612dceccd295c7c4901cd222a0d36a00064288a08b7fa5bb0a1d6046f02 WatchSource:0}: Error finding container 76677612dceccd295c7c4901cd222a0d36a00064288a08b7fa5bb0a1d6046f02: Status 404 returned error can't find the container with id 76677612dceccd295c7c4901cd222a0d36a00064288a08b7fa5bb0a1d6046f02
Sep 16 18:45:52.064307 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:52.064271    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d" event={"ID":"60248414-4995-4f51-b233-5dd9f38968a8","Type":"ContainerStarted","Data":"8ea9b950df8b7e6b48426a9f8316bfba50d528764499ea778a87ed1243de73ff"}
Sep 16 18:45:52.064454 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:52.064321    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d" event={"ID":"60248414-4995-4f51-b233-5dd9f38968a8","Type":"ContainerStarted","Data":"76677612dceccd295c7c4901cd222a0d36a00064288a08b7fa5bb0a1d6046f02"}
Sep 16 18:45:52.092394 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:52.092339    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d" podStartSLOduration=1.09232132 podStartE2EDuration="1.09232132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:52.089812475 +0000 UTC m=+1789.954716525" watchObservedRunningTime="2026-09-16 18:45:52.09232132 +0000 UTC m=+1789.957225367"
Sep 16 18:45:52.437517 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:52.437477    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:53.202295 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:53.202255    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:53.411416 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:53.411377    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"]
Sep 16 18:45:53.501223 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:53.501146    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:53.849477 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:53.849376    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:53.951182 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:53.951143    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:54.052853 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:54.052818    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr"]
Sep 16 18:45:54.059060 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:54.059025    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-8edc30-8lmfr"]
Sep 16 18:45:54.069401 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:54.069373    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:54.474114 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:54.474075    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:54.552026 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:54.551983    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b55ad86-13ad-427a-8681-f2e5032cb22c" path="/var/lib/kubelet/pods/1b55ad86-13ad-427a-8681-f2e5032cb22c/volumes"
Sep 16 18:45:54.576298 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:54.576266    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:54.619138 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:54.619101    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:45:54.673270 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:54.673215    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:55.350649 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:55.350559    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:56.224742 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:56.224703    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:56.555213 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:56.555117    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:56.989155 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:56.989118    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:57.704083 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:57.704013    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:59.413731 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:59.413691    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"]
Sep 16 18:45:59.510173 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:59.510141    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:59.583346 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:45:59.583311    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:01.649399 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:01.649365    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:46:01.649399 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:01.649410    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:46:01.662386 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:01.662361    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:46:02.140637 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:02.140606    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:46:02.703567 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:02.703525    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:46:02.706976 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:02.706951    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:46:02.710315 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:02.710294    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:46:02.713153 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:02.713136    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:46:02.930393 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:02.930359    2562 scope.go:117] "RemoveContainer" containerID="12bbfe10cc69b29f37ab0483cd447f6d697edf1f7f58d748f60a1ec0cf07e477"
Sep 16 18:46:05.628739 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:05.628704    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:08.431158 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.431116    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk" podUID="44ce8be2-bfbe-4b19-ad04-59f26e1a3603" containerName="main" containerID="cri-o://9727d4981bc5d9d9dd54e124b6b93b5cced2cf27cdfb91e67af8a86bf5583ee1" gracePeriod=45
Sep 16 18:46:08.692273 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.692214    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:46:08.767456 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.767428    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-model-cache\") pod \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") "
Sep 16 18:46:08.767661 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.767491    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-tls-certs\") pod \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") "
Sep 16 18:46:08.767661 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.767516    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-home\") pod \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") "
Sep 16 18:46:08.767784 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.767687    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-dshm\") pod \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") "
Sep 16 18:46:08.767841 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.767780    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-model-cache" (OuterVolumeSpecName: "model-cache") pod "44ce8be2-bfbe-4b19-ad04-59f26e1a3603" (UID: "44ce8be2-bfbe-4b19-ad04-59f26e1a3603"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:08.767841 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.767795    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-home" (OuterVolumeSpecName: "home") pod "44ce8be2-bfbe-4b19-ad04-59f26e1a3603" (UID: "44ce8be2-bfbe-4b19-ad04-59f26e1a3603"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:08.767841 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.767804    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkj28\" (UniqueName: \"kubernetes.io/projected/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-kube-api-access-lkj28\") pod \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") "
Sep 16 18:46:08.767841 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.767832    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-tmp-dir\") pod \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\" (UID: \"44ce8be2-bfbe-4b19-ad04-59f26e1a3603\") "
Sep 16 18:46:08.768165 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.768137    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44ce8be2-bfbe-4b19-ad04-59f26e1a3603" (UID: "44ce8be2-bfbe-4b19-ad04-59f26e1a3603"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:08.768298 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.768255    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:08.768298 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.768275    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:08.768298 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.768288    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:08.769968 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.769942    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44ce8be2-bfbe-4b19-ad04-59f26e1a3603" (UID: "44ce8be2-bfbe-4b19-ad04-59f26e1a3603"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:08.770067 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.769943    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-dshm" (OuterVolumeSpecName: "dshm") pod "44ce8be2-bfbe-4b19-ad04-59f26e1a3603" (UID: "44ce8be2-bfbe-4b19-ad04-59f26e1a3603"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:08.770067 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.769978    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-kube-api-access-lkj28" (OuterVolumeSpecName: "kube-api-access-lkj28") pod "44ce8be2-bfbe-4b19-ad04-59f26e1a3603" (UID: "44ce8be2-bfbe-4b19-ad04-59f26e1a3603"). InnerVolumeSpecName "kube-api-access-lkj28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:08.869451 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.869416    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:08.869451 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.869448    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:08.869451 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:08.869459    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkj28\" (UniqueName: \"kubernetes.io/projected/44ce8be2-bfbe-4b19-ad04-59f26e1a3603-kube-api-access-lkj28\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:09.179911 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:09.179874    2562 generic.go:356] "Generic (PLEG): container finished" podID="44ce8be2-bfbe-4b19-ad04-59f26e1a3603" containerID="9727d4981bc5d9d9dd54e124b6b93b5cced2cf27cdfb91e67af8a86bf5583ee1" exitCode=0
Sep 16 18:46:09.180123 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:09.179920    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"
Sep 16 18:46:09.180123 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:09.179958    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk" event={"ID":"44ce8be2-bfbe-4b19-ad04-59f26e1a3603","Type":"ContainerDied","Data":"9727d4981bc5d9d9dd54e124b6b93b5cced2cf27cdfb91e67af8a86bf5583ee1"}
Sep 16 18:46:09.180123 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:09.180002    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk" event={"ID":"44ce8be2-bfbe-4b19-ad04-59f26e1a3603","Type":"ContainerDied","Data":"11267c4bc6df90bbd30d86d7c5bb54f4e7b6db31a656b1dc3542f5dbc1d1013e"}
Sep 16 18:46:09.180123 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:09.180021    2562 scope.go:117] "RemoveContainer" containerID="9727d4981bc5d9d9dd54e124b6b93b5cced2cf27cdfb91e67af8a86bf5583ee1"
Sep 16 18:46:09.190789 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:09.190768    2562 scope.go:117] "RemoveContainer" containerID="9727d4981bc5d9d9dd54e124b6b93b5cced2cf27cdfb91e67af8a86bf5583ee1"
Sep 16 18:46:09.191129 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:46:09.191112    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9727d4981bc5d9d9dd54e124b6b93b5cced2cf27cdfb91e67af8a86bf5583ee1\": container with ID starting with 9727d4981bc5d9d9dd54e124b6b93b5cced2cf27cdfb91e67af8a86bf5583ee1 not found: ID does not exist" containerID="9727d4981bc5d9d9dd54e124b6b93b5cced2cf27cdfb91e67af8a86bf5583ee1"
Sep 16 18:46:09.191198 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:09.191140    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9727d4981bc5d9d9dd54e124b6b93b5cced2cf27cdfb91e67af8a86bf5583ee1"} err="failed to get container status \"9727d4981bc5d9d9dd54e124b6b93b5cced2cf27cdfb91e67af8a86bf5583ee1\": rpc error: code = NotFound desc = could not find container \"9727d4981bc5d9d9dd54e124b6b93b5cced2cf27cdfb91e67af8a86bf5583ee1\": container with ID starting with 9727d4981bc5d9d9dd54e124b6b93b5cced2cf27cdfb91e67af8a86bf5583ee1 not found: ID does not exist"
Sep 16 18:46:09.202819 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:09.202790    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"]
Sep 16 18:46:09.207938 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:09.207911    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ab83f76b/e2e-lifecycle-model-ab83f76b-kserve-67f4ff8f8c-kjtsk"]
Sep 16 18:46:10.551891 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:10.551854    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44ce8be2-bfbe-4b19-ad04-59f26e1a3603" path="/var/lib/kubelet/pods/44ce8be2-bfbe-4b19-ad04-59f26e1a3603/volumes"
Sep 16 18:46:14.433564 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.433508    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7" podUID="48453f2d-c721-4699-ab46-0c713911176d" containerName="main" containerID="cri-o://c3f63aa1c602bfd463a1a14a9b4b047e9a82afaa40ca8f4951504b8e193ec141" gracePeriod=45
Sep 16 18:46:14.693868 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.693797    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:46:14.828189 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.828156    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-dshm\") pod \"48453f2d-c721-4699-ab46-0c713911176d\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") "
Sep 16 18:46:14.828370 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.828205    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-model-cache\") pod \"48453f2d-c721-4699-ab46-0c713911176d\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") "
Sep 16 18:46:14.828370 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.828236    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqzv5\" (UniqueName: \"kubernetes.io/projected/48453f2d-c721-4699-ab46-0c713911176d-kube-api-access-jqzv5\") pod \"48453f2d-c721-4699-ab46-0c713911176d\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") "
Sep 16 18:46:14.828370 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.828333    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48453f2d-c721-4699-ab46-0c713911176d-tls-certs\") pod \"48453f2d-c721-4699-ab46-0c713911176d\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") "
Sep 16 18:46:14.828505 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.828388    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-tmp-dir\") pod \"48453f2d-c721-4699-ab46-0c713911176d\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") "
Sep 16 18:46:14.828565 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.828500    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-model-cache" (OuterVolumeSpecName: "model-cache") pod "48453f2d-c721-4699-ab46-0c713911176d" (UID: "48453f2d-c721-4699-ab46-0c713911176d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:14.828565 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.828523    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-home\") pod \"48453f2d-c721-4699-ab46-0c713911176d\" (UID: \"48453f2d-c721-4699-ab46-0c713911176d\") "
Sep 16 18:46:14.828771 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.828749    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-home" (OuterVolumeSpecName: "home") pod "48453f2d-c721-4699-ab46-0c713911176d" (UID: "48453f2d-c721-4699-ab46-0c713911176d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:14.828924 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.828899    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48453f2d-c721-4699-ab46-0c713911176d" (UID: "48453f2d-c721-4699-ab46-0c713911176d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:14.829067 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.828973    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:14.829067 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.828988    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:14.829067 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.829001    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:14.830530 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.830501    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-dshm" (OuterVolumeSpecName: "dshm") pod "48453f2d-c721-4699-ab46-0c713911176d" (UID: "48453f2d-c721-4699-ab46-0c713911176d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:14.830647 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.830516    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48453f2d-c721-4699-ab46-0c713911176d-kube-api-access-jqzv5" (OuterVolumeSpecName: "kube-api-access-jqzv5") pod "48453f2d-c721-4699-ab46-0c713911176d" (UID: "48453f2d-c721-4699-ab46-0c713911176d"). InnerVolumeSpecName "kube-api-access-jqzv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:14.830647 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.830586    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48453f2d-c721-4699-ab46-0c713911176d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48453f2d-c721-4699-ab46-0c713911176d" (UID: "48453f2d-c721-4699-ab46-0c713911176d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:14.930089 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.930050    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48453f2d-c721-4699-ab46-0c713911176d-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:14.930089 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.930085    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqzv5\" (UniqueName: \"kubernetes.io/projected/48453f2d-c721-4699-ab46-0c713911176d-kube-api-access-jqzv5\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:14.930286 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:14.930098    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48453f2d-c721-4699-ab46-0c713911176d-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:15.218886 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:15.218846    2562 generic.go:356] "Generic (PLEG): container finished" podID="48453f2d-c721-4699-ab46-0c713911176d" containerID="c3f63aa1c602bfd463a1a14a9b4b047e9a82afaa40ca8f4951504b8e193ec141" exitCode=0
Sep 16 18:46:15.219077 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:15.218919    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"
Sep 16 18:46:15.219077 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:15.218935    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7" event={"ID":"48453f2d-c721-4699-ab46-0c713911176d","Type":"ContainerDied","Data":"c3f63aa1c602bfd463a1a14a9b4b047e9a82afaa40ca8f4951504b8e193ec141"}
Sep 16 18:46:15.219077 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:15.218984    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7" event={"ID":"48453f2d-c721-4699-ab46-0c713911176d","Type":"ContainerDied","Data":"a11aba6c1e233681197a8b590d1189bebd7e196378de6c3e0710ecc9549d280e"}
Sep 16 18:46:15.219077 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:15.219003    2562 scope.go:117] "RemoveContainer" containerID="c3f63aa1c602bfd463a1a14a9b4b047e9a82afaa40ca8f4951504b8e193ec141"
Sep 16 18:46:15.229460 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:15.229442    2562 scope.go:117] "RemoveContainer" containerID="c3f63aa1c602bfd463a1a14a9b4b047e9a82afaa40ca8f4951504b8e193ec141"
Sep 16 18:46:15.229806 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:46:15.229787    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f63aa1c602bfd463a1a14a9b4b047e9a82afaa40ca8f4951504b8e193ec141\": container with ID starting with c3f63aa1c602bfd463a1a14a9b4b047e9a82afaa40ca8f4951504b8e193ec141 not found: ID does not exist" containerID="c3f63aa1c602bfd463a1a14a9b4b047e9a82afaa40ca8f4951504b8e193ec141"
Sep 16 18:46:15.229868 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:15.229815    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f63aa1c602bfd463a1a14a9b4b047e9a82afaa40ca8f4951504b8e193ec141"} err="failed to get container status \"c3f63aa1c602bfd463a1a14a9b4b047e9a82afaa40ca8f4951504b8e193ec141\": rpc error: code = NotFound desc = could not find container \"c3f63aa1c602bfd463a1a14a9b4b047e9a82afaa40ca8f4951504b8e193ec141\": container with ID starting with c3f63aa1c602bfd463a1a14a9b4b047e9a82afaa40ca8f4951504b8e193ec141 not found: ID does not exist"
Sep 16 18:46:15.242820 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:15.242787    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"]
Sep 16 18:46:15.245338 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:15.245316    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8edc30/auth-test-model-8edc30-kserve-86f8cb9b7f-f8fg7"]
Sep 16 18:46:15.412782 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:15.412745    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:46:16.550477 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:16.550435    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48453f2d-c721-4699-ab46-0c713911176d" path="/var/lib/kubelet/pods/48453f2d-c721-4699-ab46-0c713911176d/volumes"
Sep 16 18:46:29.724663 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:29.724628    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:31.244245 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:31.244214    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:36.985817 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:36.985780    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:37.772697 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:37.772665    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:39.830807 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.830768    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"]
Sep 16 18:46:39.831423 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.831407    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48453f2d-c721-4699-ab46-0c713911176d" containerName="main"
Sep 16 18:46:39.831470 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.831427    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="48453f2d-c721-4699-ab46-0c713911176d" containerName="main"
Sep 16 18:46:39.831470 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.831454    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ce8be2-bfbe-4b19-ad04-59f26e1a3603" containerName="main"
Sep 16 18:46:39.831470 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.831464    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ce8be2-bfbe-4b19-ad04-59f26e1a3603" containerName="main"
Sep 16 18:46:39.831613 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.831594    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ce8be2-bfbe-4b19-ad04-59f26e1a3603" containerName="main"
Sep 16 18:46:39.831613 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.831610    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="48453f2d-c721-4699-ab46-0c713911176d" containerName="main"
Sep 16 18:46:39.835018 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.834996    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.842728 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.842704    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d8a600\"/\"distinct-2-w4-d8a600-kserve-self-signed-certs\""
Sep 16 18:46:39.845188 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.845157    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"]
Sep 16 18:46:39.871410 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.871381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-dshm\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.871618 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.871421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b876aa39-20ce-49c3-9d52-684d4590bede-tls-certs\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.871618 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.871455    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbbm9\" (UniqueName: \"kubernetes.io/projected/b876aa39-20ce-49c3-9d52-684d4590bede-kube-api-access-mbbm9\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.871618 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.871513    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-home\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.871799 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.871635    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-model-cache\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.871799 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.871661    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-tmp-dir\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.973162 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.972695    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-model-cache\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.973421 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.973398    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-tmp-dir\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.973672 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.973652    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-dshm\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.973810 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.973794    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b876aa39-20ce-49c3-9d52-684d4590bede-tls-certs\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.973941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.973923    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbbm9\" (UniqueName: \"kubernetes.io/projected/b876aa39-20ce-49c3-9d52-684d4590bede-kube-api-access-mbbm9\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.974103 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.973136    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-model-cache\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.974617 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.974092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-home\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.974845 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.974357    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-home\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.974983 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.973741    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-tmp-dir\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.978393 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.978221    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b876aa39-20ce-49c3-9d52-684d4590bede-tls-certs\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.978393 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.978335    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-dshm\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:39.983782 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:39.983725    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbbm9\" (UniqueName: \"kubernetes.io/projected/b876aa39-20ce-49c3-9d52-684d4590bede-kube-api-access-mbbm9\") pod \"distinct-2-w4-d8a600-kserve-755487895-hckwj\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") " pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:40.092576 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:40.092476    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:40.145559 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:40.145515    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:40.437662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:40.437637    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"]
Sep 16 18:46:40.440439 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:46:40.440406    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb876aa39_20ce_49c3_9d52_684d4590bede.slice/crio-32c07c1293f46731fe8ce90e3446d744c9bf5f800387141c6168fad6713cd1d7 WatchSource:0}: Error finding container 32c07c1293f46731fe8ce90e3446d744c9bf5f800387141c6168fad6713cd1d7: Status 404 returned error can't find the container with id 32c07c1293f46731fe8ce90e3446d744c9bf5f800387141c6168fad6713cd1d7
Sep 16 18:46:41.394375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:41.394337    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj" event={"ID":"b876aa39-20ce-49c3-9d52-684d4590bede","Type":"ContainerStarted","Data":"405a9874ab696f40c124ddd648e4916771de7213e5f08c142be27b16a8689512"}
Sep 16 18:46:41.394807 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:41.394385    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj" event={"ID":"b876aa39-20ce-49c3-9d52-684d4590bede","Type":"ContainerStarted","Data":"32c07c1293f46731fe8ce90e3446d744c9bf5f800387141c6168fad6713cd1d7"}
Sep 16 18:46:41.413759 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:41.413707    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj" podStartSLOduration=2.413691183 podStartE2EDuration="2.413691183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:41.4110633 +0000 UTC m=+1839.275967365" watchObservedRunningTime="2026-09-16 18:46:41.413691183 +0000 UTC m=+1839.278595230"
Sep 16 18:46:43.998504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:43.998464    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:44.100203 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:44.100140    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:44.317302 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:44.317210    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:44.703395 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:44.703358    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b"]
Sep 16 18:46:44.707072 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:44.707054    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b"
Sep 16 18:46:44.710970 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:44.710947    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:46:44.712136 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:44.712116    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:46:44.743986 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:44.743957    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b"]
Sep 16 18:46:44.824627 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:44.824587    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/105b6ca4-aa46-43e3-9a08-1eebc98e10f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b\" (UID: \"105b6ca4-aa46-43e3-9a08-1eebc98e10f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b"
Sep 16 18:46:44.926098 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:44.926055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/105b6ca4-aa46-43e3-9a08-1eebc98e10f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b\" (UID: \"105b6ca4-aa46-43e3-9a08-1eebc98e10f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b"
Sep 16 18:46:44.926716 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:44.926698    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/105b6ca4-aa46-43e3-9a08-1eebc98e10f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b\" (UID: \"105b6ca4-aa46-43e3-9a08-1eebc98e10f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b"
Sep 16 18:46:45.018606 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:45.018510    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b"
Sep 16 18:46:45.099196 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:45.099166    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:45.308239 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:45.308209    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b"]
Sep 16 18:46:45.309325 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:46:45.309294    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105b6ca4_aa46_43e3_9a08_1eebc98e10f4.slice/crio-cc2b17af49f8a988378235973946b843b8825c963597e69062e6200cb038e89e WatchSource:0}: Error finding container cc2b17af49f8a988378235973946b843b8825c963597e69062e6200cb038e89e: Status 404 returned error can't find the container with id cc2b17af49f8a988378235973946b843b8825c963597e69062e6200cb038e89e
Sep 16 18:46:45.424654 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:45.424619    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b" event={"ID":"105b6ca4-aa46-43e3-9a08-1eebc98e10f4","Type":"ContainerStarted","Data":"cc2b17af49f8a988378235973946b843b8825c963597e69062e6200cb038e89e"}
Sep 16 18:46:45.856829 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:45.856796    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:46:46.166133 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:46.166061    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:46.433030 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:46.432957    2562 generic.go:356] "Generic (PLEG): container finished" podID="105b6ca4-aa46-43e3-9a08-1eebc98e10f4" containerID="b5daa2eaf058f11ef8ce46f120576cd89a7197201f0051bf3b13480f192e4436" exitCode=0
Sep 16 18:46:46.433174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:46.433036    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b" event={"ID":"105b6ca4-aa46-43e3-9a08-1eebc98e10f4","Type":"ContainerDied","Data":"b5daa2eaf058f11ef8ce46f120576cd89a7197201f0051bf3b13480f192e4436"}
Sep 16 18:46:47.158217 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:47.158182    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:47.579111 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:47.579083    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b"
Sep 16 18:46:47.654341 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:47.654305    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/105b6ca4-aa46-43e3-9a08-1eebc98e10f4-maas-api-service-ca\") pod \"105b6ca4-aa46-43e3-9a08-1eebc98e10f4\" (UID: \"105b6ca4-aa46-43e3-9a08-1eebc98e10f4\") "
Sep 16 18:46:47.654756 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:47.654717    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105b6ca4-aa46-43e3-9a08-1eebc98e10f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "105b6ca4-aa46-43e3-9a08-1eebc98e10f4" (UID: "105b6ca4-aa46-43e3-9a08-1eebc98e10f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:47.755783 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:47.755742    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/105b6ca4-aa46-43e3-9a08-1eebc98e10f4-maas-api-service-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:48.011387 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:48.011293    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:48.460324 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:48.460252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b" event={"ID":"105b6ca4-aa46-43e3-9a08-1eebc98e10f4","Type":"ContainerDied","Data":"cc2b17af49f8a988378235973946b843b8825c963597e69062e6200cb038e89e"}
Sep 16 18:46:48.460520 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:48.460375    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b"
Sep 16 18:46:48.460520 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:48.460384    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc2b17af49f8a988378235973946b843b8825c963597e69062e6200cb038e89e"
Sep 16 18:46:48.851098 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:48.850996    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:49.298030 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:49.297998    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:49.393961 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:49.393925    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:49.547486 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:49.547445    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"]
Sep 16 18:46:49.547723 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:49.547695    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb" podUID="df4a86ae-3905-4322-8108-c6780fd87ac8" containerName="istio-proxy" containerID="cri-o://73badb4f85f1770ea5fab00a95559840b6a113fa155c2522ad7c3a2d696e9fb9" gracePeriod=30
Sep 16 18:46:49.569272 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:49.569235    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:49.844647 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:49.844508    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:50.146514 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:50.146411    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:50.146955 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:50.146670    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:50.159336 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:50.159307    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:50.487946 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:50.487916    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:46:50.917259 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:50.917168    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.022218 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:51.022187    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.122815 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:51.122781    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.233465 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:51.233429    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.832318 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:51.832273    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:51.953135 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:51.953103    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:53.319206 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:53.319168    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:54.767027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.766987    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:54.825915 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.825854    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:46:54.830713 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.830679    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:54.927690 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.927654    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-workload-certs\") pod \"df4a86ae-3905-4322-8108-c6780fd87ac8\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") "
Sep 16 18:46:54.927878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.927703    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6kx6\" (UniqueName: \"kubernetes.io/projected/df4a86ae-3905-4322-8108-c6780fd87ac8-kube-api-access-r6kx6\") pod \"df4a86ae-3905-4322-8108-c6780fd87ac8\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") "
Sep 16 18:46:54.927878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.927784    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-podinfo\") pod \"df4a86ae-3905-4322-8108-c6780fd87ac8\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") "
Sep 16 18:46:54.927878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.927868    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-token\") pod \"df4a86ae-3905-4322-8108-c6780fd87ac8\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") "
Sep 16 18:46:54.928049 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.927898    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-credential-socket\") pod \"df4a86ae-3905-4322-8108-c6780fd87ac8\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") "
Sep 16 18:46:54.928049 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.927934    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4a86ae-3905-4322-8108-c6780fd87ac8-istiod-ca-cert\") pod \"df4a86ae-3905-4322-8108-c6780fd87ac8\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") "
Sep 16 18:46:54.928049 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.927934    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "df4a86ae-3905-4322-8108-c6780fd87ac8" (UID: "df4a86ae-3905-4322-8108-c6780fd87ac8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:54.928049 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.927970    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-workload-socket\") pod \"df4a86ae-3905-4322-8108-c6780fd87ac8\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") "
Sep 16 18:46:54.928049 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.927998    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-data\") pod \"df4a86ae-3905-4322-8108-c6780fd87ac8\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") "
Sep 16 18:46:54.928049 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.928033    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-envoy\") pod \"df4a86ae-3905-4322-8108-c6780fd87ac8\" (UID: \"df4a86ae-3905-4322-8108-c6780fd87ac8\") "
Sep 16 18:46:54.928365 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.928156    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "df4a86ae-3905-4322-8108-c6780fd87ac8" (UID: "df4a86ae-3905-4322-8108-c6780fd87ac8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:54.928365 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.928263    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4a86ae-3905-4322-8108-c6780fd87ac8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "df4a86ae-3905-4322-8108-c6780fd87ac8" (UID: "df4a86ae-3905-4322-8108-c6780fd87ac8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:54.928365 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.928337    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "df4a86ae-3905-4322-8108-c6780fd87ac8" (UID: "df4a86ae-3905-4322-8108-c6780fd87ac8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:54.928620 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.928407    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-data" (OuterVolumeSpecName: "istio-data") pod "df4a86ae-3905-4322-8108-c6780fd87ac8" (UID: "df4a86ae-3905-4322-8108-c6780fd87ac8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:54.928620 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.928439    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-workload-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:54.928620 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.928468    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-credential-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:54.928620 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.928481    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4a86ae-3905-4322-8108-c6780fd87ac8-istiod-ca-cert\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:54.930176 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.930141    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "df4a86ae-3905-4322-8108-c6780fd87ac8" (UID: "df4a86ae-3905-4322-8108-c6780fd87ac8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:46:54.930305 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.930221    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-token" (OuterVolumeSpecName: "istio-token") pod "df4a86ae-3905-4322-8108-c6780fd87ac8" (UID: "df4a86ae-3905-4322-8108-c6780fd87ac8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:54.930512 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.930484    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4a86ae-3905-4322-8108-c6780fd87ac8-kube-api-access-r6kx6" (OuterVolumeSpecName: "kube-api-access-r6kx6") pod "df4a86ae-3905-4322-8108-c6780fd87ac8" (UID: "df4a86ae-3905-4322-8108-c6780fd87ac8"). InnerVolumeSpecName "kube-api-access-r6kx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:54.930512 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.930490    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "df4a86ae-3905-4322-8108-c6780fd87ac8" (UID: "df4a86ae-3905-4322-8108-c6780fd87ac8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:54.966469 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:54.966439    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:55.029291 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.029247    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-token\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:55.029291 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.029288    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-workload-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:55.029575 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.029302    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-data\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:55.029575 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.029315    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-envoy\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:55.029575 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.029330    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6kx6\" (UniqueName: \"kubernetes.io/projected/df4a86ae-3905-4322-8108-c6780fd87ac8-kube-api-access-r6kx6\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:55.029575 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.029345    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4a86ae-3905-4322-8108-c6780fd87ac8-istio-podinfo\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:46:55.510512 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.510474    2562 generic.go:356] "Generic (PLEG): container finished" podID="df4a86ae-3905-4322-8108-c6780fd87ac8" containerID="73badb4f85f1770ea5fab00a95559840b6a113fa155c2522ad7c3a2d696e9fb9" exitCode=0
Sep 16 18:46:55.510725 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.510532    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"
Sep 16 18:46:55.510725 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.510581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb" event={"ID":"df4a86ae-3905-4322-8108-c6780fd87ac8","Type":"ContainerDied","Data":"73badb4f85f1770ea5fab00a95559840b6a113fa155c2522ad7c3a2d696e9fb9"}
Sep 16 18:46:55.510725 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.510650    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb" event={"ID":"df4a86ae-3905-4322-8108-c6780fd87ac8","Type":"ContainerDied","Data":"a83b814bb5c0edc5de088b7732f28068407783d4781bb885bec9f4cb3143e3ca"}
Sep 16 18:46:55.510725 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.510680    2562 scope.go:117] "RemoveContainer" containerID="73badb4f85f1770ea5fab00a95559840b6a113fa155c2522ad7c3a2d696e9fb9"
Sep 16 18:46:55.520839 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.520804    2562 scope.go:117] "RemoveContainer" containerID="406c16239f7c8de63ee9736f237786e62c92d50b73055a69b2fae925b68d82bb"
Sep 16 18:46:55.530153 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.530134    2562 scope.go:117] "RemoveContainer" containerID="73badb4f85f1770ea5fab00a95559840b6a113fa155c2522ad7c3a2d696e9fb9"
Sep 16 18:46:55.530446 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:46:55.530426    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73badb4f85f1770ea5fab00a95559840b6a113fa155c2522ad7c3a2d696e9fb9\": container with ID starting with 73badb4f85f1770ea5fab00a95559840b6a113fa155c2522ad7c3a2d696e9fb9 not found: ID does not exist" containerID="73badb4f85f1770ea5fab00a95559840b6a113fa155c2522ad7c3a2d696e9fb9"
Sep 16 18:46:55.530517 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.530455    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73badb4f85f1770ea5fab00a95559840b6a113fa155c2522ad7c3a2d696e9fb9"} err="failed to get container status \"73badb4f85f1770ea5fab00a95559840b6a113fa155c2522ad7c3a2d696e9fb9\": rpc error: code = NotFound desc = could not find container \"73badb4f85f1770ea5fab00a95559840b6a113fa155c2522ad7c3a2d696e9fb9\": container with ID starting with 73badb4f85f1770ea5fab00a95559840b6a113fa155c2522ad7c3a2d696e9fb9 not found: ID does not exist"
Sep 16 18:46:55.530517 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.530473    2562 scope.go:117] "RemoveContainer" containerID="406c16239f7c8de63ee9736f237786e62c92d50b73055a69b2fae925b68d82bb"
Sep 16 18:46:55.530748 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:46:55.530730    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406c16239f7c8de63ee9736f237786e62c92d50b73055a69b2fae925b68d82bb\": container with ID starting with 406c16239f7c8de63ee9736f237786e62c92d50b73055a69b2fae925b68d82bb not found: ID does not exist" containerID="406c16239f7c8de63ee9736f237786e62c92d50b73055a69b2fae925b68d82bb"
Sep 16 18:46:55.530790 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.530754    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406c16239f7c8de63ee9736f237786e62c92d50b73055a69b2fae925b68d82bb"} err="failed to get container status \"406c16239f7c8de63ee9736f237786e62c92d50b73055a69b2fae925b68d82bb\": rpc error: code = NotFound desc = could not find container \"406c16239f7c8de63ee9736f237786e62c92d50b73055a69b2fae925b68d82bb\": container with ID starting with 406c16239f7c8de63ee9736f237786e62c92d50b73055a69b2fae925b68d82bb not found: ID does not exist"
Sep 16 18:46:55.538769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.538748    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"]
Sep 16 18:46:55.542578 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.542556    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ab83f76b-openshift-default-c8d959f6-zjrrb"]
Sep 16 18:46:55.749993 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.749952    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:55.763835 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.763762    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ee3af0-29826405-l672n"]
Sep 16 18:46:55.768034 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.768009    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ee3af0-29826405-l672n"]
Sep 16 18:46:55.854691 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.854637    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:55.955938 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:55.955901    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:56.080128 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:56.080046    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:56.132806 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:56.132771    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b"]
Sep 16 18:46:56.154571 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:56.152762    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ee3af0-hnh5b"]
Sep 16 18:46:56.171311 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:56.171272    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:56.472455 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:56.472422    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:56.550264 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:56.550228    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="105b6ca4-aa46-43e3-9a08-1eebc98e10f4" path="/var/lib/kubelet/pods/105b6ca4-aa46-43e3-9a08-1eebc98e10f4/volumes"
Sep 16 18:46:56.550698 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:56.550677    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d0c0806-c126-4ba0-a1af-f24878c45924" path="/var/lib/kubelet/pods/7d0c0806-c126-4ba0-a1af-f24878c45924/volumes"
Sep 16 18:46:56.551114 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:56.551096    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df4a86ae-3905-4322-8108-c6780fd87ac8" path="/var/lib/kubelet/pods/df4a86ae-3905-4322-8108-c6780fd87ac8/volumes"
Sep 16 18:46:56.576332 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:56.576303    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:57.584826 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:57.584790    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:58.717791 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:58.717754    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.293196 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:59.293158    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:59.548658 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:46:59.548495    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:00.668511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:00.668472    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:00.743036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:00.743001    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:01.333528 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:01.333495    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:02.960982 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:02.960948    2562 scope.go:117] "RemoveContainer" containerID="24f9d253e1fc34561f1bce315b290ae85a03838795a9063e982367e07d4732e8"
Sep 16 18:47:02.968579 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:02.968518    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:03.092586 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:03.092539    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:03.414401 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:03.414310    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"]
Sep 16 18:47:03.414616 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:03.414590    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952" podUID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" containerName="istio-proxy" containerID="cri-o://d5f19ce291c1c6c3499974ab1b1d25cedbca0c4f77ec8da40261fdfab2b45a11" gracePeriod=30
Sep 16 18:47:03.428122 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:03.428097    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:04.644970 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:04.644931    2562 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 18:47:04.645343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:04.644993    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952" podUID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:47:08.576847 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.576813    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:08.601511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.601481    2562 generic.go:356] "Generic (PLEG): container finished" podID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" containerID="d5f19ce291c1c6c3499974ab1b1d25cedbca0c4f77ec8da40261fdfab2b45a11" exitCode=0
Sep 16 18:47:08.601683 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.601558    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952" event={"ID":"4d9450ef-fcd8-467d-9be6-28b3d67d79f8","Type":"ContainerDied","Data":"d5f19ce291c1c6c3499974ab1b1d25cedbca0c4f77ec8da40261fdfab2b45a11"}
Sep 16 18:47:08.601683 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.601625    2562 scope.go:117] "RemoveContainer" containerID="8ef5146b7963eaafb69ae0ccafe0a02c5ec94aec4b9bd73497f2c6c552f0093a"
Sep 16 18:47:08.680572 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.680485    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:47:08.704395 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.704362    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:08.767117 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.767088    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-workload-socket\") pod \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") "
Sep 16 18:47:08.767286 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.767127    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-podinfo\") pod \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") "
Sep 16 18:47:08.767286 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.767151    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-envoy\") pod \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") "
Sep 16 18:47:08.767286 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.767176    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-token\") pod \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") "
Sep 16 18:47:08.767463 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.767329    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-workload-certs\") pod \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") "
Sep 16 18:47:08.767463 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.767400    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-data\") pod \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") "
Sep 16 18:47:08.767463 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.767449    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4d9450ef-fcd8-467d-9be6-28b3d67d79f8" (UID: "4d9450ef-fcd8-467d-9be6-28b3d67d79f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:08.767653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.767458    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phmr6\" (UniqueName: \"kubernetes.io/projected/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-kube-api-access-phmr6\") pod \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") "
Sep 16 18:47:08.767653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.767601    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4d9450ef-fcd8-467d-9be6-28b3d67d79f8" (UID: "4d9450ef-fcd8-467d-9be6-28b3d67d79f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:08.767653 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.767620    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-credential-socket\") pod \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") "
Sep 16 18:47:08.767825 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.767663    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istiod-ca-cert\") pod \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\" (UID: \"4d9450ef-fcd8-467d-9be6-28b3d67d79f8\") "
Sep 16 18:47:08.767825 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.767733    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "4d9450ef-fcd8-467d-9be6-28b3d67d79f8" (UID: "4d9450ef-fcd8-467d-9be6-28b3d67d79f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:08.767935 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.767839    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4d9450ef-fcd8-467d-9be6-28b3d67d79f8" (UID: "4d9450ef-fcd8-467d-9be6-28b3d67d79f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:08.768060 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.768033    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-credential-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:47:08.768196 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.768062    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-workload-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:47:08.768196 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.768058    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4d9450ef-fcd8-467d-9be6-28b3d67d79f8" (UID: "4d9450ef-fcd8-467d-9be6-28b3d67d79f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:08.768196 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.768078    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-workload-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:47:08.768196 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.768093    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-data\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:47:08.769310 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.769284    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4d9450ef-fcd8-467d-9be6-28b3d67d79f8" (UID: "4d9450ef-fcd8-467d-9be6-28b3d67d79f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:47:08.769401 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.769358    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "4d9450ef-fcd8-467d-9be6-28b3d67d79f8" (UID: "4d9450ef-fcd8-467d-9be6-28b3d67d79f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:08.769451 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.769417    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4d9450ef-fcd8-467d-9be6-28b3d67d79f8" (UID: "4d9450ef-fcd8-467d-9be6-28b3d67d79f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:08.769602 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.769573    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-kube-api-access-phmr6" (OuterVolumeSpecName: "kube-api-access-phmr6") pod "4d9450ef-fcd8-467d-9be6-28b3d67d79f8" (UID: "4d9450ef-fcd8-467d-9be6-28b3d67d79f8"). InnerVolumeSpecName "kube-api-access-phmr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:08.795869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.795841    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:08.869377 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.869341    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phmr6\" (UniqueName: \"kubernetes.io/projected/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-kube-api-access-phmr6\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:47:08.869377 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.869369    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istiod-ca-cert\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:47:08.869377 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.869379    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-podinfo\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:47:08.869634 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.869387    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-envoy\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:47:08.869634 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:08.869394    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d9450ef-fcd8-467d-9be6-28b3d67d79f8-istio-token\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:47:09.609342 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:09.609311    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"
Sep 16 18:47:09.609342 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:09.609333    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952" event={"ID":"4d9450ef-fcd8-467d-9be6-28b3d67d79f8","Type":"ContainerDied","Data":"55dc0fdc88a0a48e56f29d723fb3222f900326cb00c904360a91d5d41892f8c1"}
Sep 16 18:47:09.609893 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:09.609390    2562 scope.go:117] "RemoveContainer" containerID="d5f19ce291c1c6c3499974ab1b1d25cedbca0c4f77ec8da40261fdfab2b45a11"
Sep 16 18:47:09.618987 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:09.618958    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:47:09.636414 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:09.636387    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"]
Sep 16 18:47:09.639878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:09.639854    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ee3af0-openshift-default-6997d6dfbb-gx952"]
Sep 16 18:47:10.550892 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:10.550853    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" path="/var/lib/kubelet/pods/4d9450ef-fcd8-467d-9be6-28b3d67d79f8/volumes"
Sep 16 18:47:13.141735 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:13.141701    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:19.368735 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:19.368698    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:19.608791 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:19.608759    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:27.513288 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:27.513193    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:47:40.525698 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:40.525661    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:46.869301 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:46.869263    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:46.967916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:46.967882    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:47.245564 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:47.244650    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:47:47.247669 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:47.247636    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:47.663867 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:47.663773    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:47.793623 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:47.793522    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:50.383317 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:50.383286    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:51.150735 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:51.150699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:51.248873 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:51.248831    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:52.080125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:52.080092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:53.023070 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:53.023034    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:53.906939 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:53.906909    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:54.014759 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:54.014723    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:54.164062 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:54.163978    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:54.877705 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:54.877674    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:55.070278 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:47:55.070248    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:03.001503 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:03.001449    2562 scope.go:117] "RemoveContainer" containerID="f7e33790d22e55a9f5c2c1a402f75e936187e6cc468cc0ce5bfc93570cecbc07"
Sep 16 18:48:11.056880 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:11.056839    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:48:11.661405 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:11.661367    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:17.024214 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:17.024179    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:18.179756 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:18.179722    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:19.103952 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:19.103919    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:19.895517 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:19.895482    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:20.693456 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:20.693425    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:21.103703 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.103665    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:21.566503 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.566465    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:21.928312 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928235    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"]
Sep 16 18:48:21.928662 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928648    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df4a86ae-3905-4322-8108-c6780fd87ac8" containerName="istio-proxy"
Sep 16 18:48:21.928713 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928664    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="df4a86ae-3905-4322-8108-c6780fd87ac8" containerName="istio-proxy"
Sep 16 18:48:21.928713 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928672    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df4a86ae-3905-4322-8108-c6780fd87ac8" containerName="istio-proxy"
Sep 16 18:48:21.928713 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928677    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="df4a86ae-3905-4322-8108-c6780fd87ac8" containerName="istio-proxy"
Sep 16 18:48:21.928713 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928691    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" containerName="istio-proxy"
Sep 16 18:48:21.928713 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928697    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" containerName="istio-proxy"
Sep 16 18:48:21.928713 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928705    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105b6ca4-aa46-43e3-9a08-1eebc98e10f4" containerName="revoke-keys"
Sep 16 18:48:21.928713 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928710    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="105b6ca4-aa46-43e3-9a08-1eebc98e10f4" containerName="revoke-keys"
Sep 16 18:48:21.928930 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928725    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" containerName="istio-proxy"
Sep 16 18:48:21.928930 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928730    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" containerName="istio-proxy"
Sep 16 18:48:21.928930 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928789    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" containerName="istio-proxy"
Sep 16 18:48:21.928930 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928797    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="df4a86ae-3905-4322-8108-c6780fd87ac8" containerName="istio-proxy"
Sep 16 18:48:21.928930 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928805    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="105b6ca4-aa46-43e3-9a08-1eebc98e10f4" containerName="revoke-keys"
Sep 16 18:48:21.928930 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928810    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="df4a86ae-3905-4322-8108-c6780fd87ac8" containerName="istio-proxy"
Sep 16 18:48:21.928930 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.928817    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9450ef-fcd8-467d-9be6-28b3d67d79f8" containerName="istio-proxy"
Sep 16 18:48:21.932129 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.932109    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:21.935093 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.935073    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4dfacb-openshift-default-dockercfg-2qt42\""
Sep 16 18:48:21.969916 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:21.969885    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"]
Sep 16 18:48:22.009313 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.009276    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:22.095131 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.095101    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-podinfo\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.095312 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.095152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxxnv\" (UniqueName: \"kubernetes.io/projected/f64bc0a9-db3a-4e00-90db-4d81098629b2-kube-api-access-sxxnv\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.095312 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.095220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-workload-certs\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.095312 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.095259    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-workload-socket\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.095312 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.095294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f64bc0a9-db3a-4e00-90db-4d81098629b2-istiod-ca-cert\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.095457 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.095352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-data\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.095457 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.095416    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-credential-socket\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.095457 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.095441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-token\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.095577 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.095480    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-envoy\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.196595 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.196474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxxnv\" (UniqueName: \"kubernetes.io/projected/f64bc0a9-db3a-4e00-90db-4d81098629b2-kube-api-access-sxxnv\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.196595 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.196528    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-workload-certs\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.196595 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.196579    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-workload-socket\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.197163 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.196612    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f64bc0a9-db3a-4e00-90db-4d81098629b2-istiod-ca-cert\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.197163 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.196651    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-data\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.197163 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.196731    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-credential-socket\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.197163 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.196770    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-token\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.197163 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.196803    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-envoy\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.197163 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.196841    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-podinfo\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.197163 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.197022    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-workload-certs\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.197163 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.197158    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-data\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.197518 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.197177    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-credential-socket\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.197518 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.197256    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-workload-socket\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.197518 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.197347    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f64bc0a9-db3a-4e00-90db-4d81098629b2-istiod-ca-cert\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.199051 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.199030    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-envoy\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.199248 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.199231    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-podinfo\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.205616 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.205591    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxxnv\" (UniqueName: \"kubernetes.io/projected/f64bc0a9-db3a-4e00-90db-4d81098629b2-kube-api-access-sxxnv\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.205734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.205643    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-token\") pod \"e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") " pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.242528 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.242504    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:22.414440 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.414404    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:22.526387 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.526304    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"]
Sep 16 18:48:22.528914 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:48:22.528885    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64bc0a9_db3a_4e00_90db_4d81098629b2.slice/crio-c5e8f0ee9777e24752f243b04319082d9b25068c9ba28f43fc2410de60e63333 WatchSource:0}: Error finding container c5e8f0ee9777e24752f243b04319082d9b25068c9ba28f43fc2410de60e63333: Status 404 returned error can't find the container with id c5e8f0ee9777e24752f243b04319082d9b25068c9ba28f43fc2410de60e63333
Sep 16 18:48:22.531331 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.531299    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:48:22.531439 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.531380    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:48:22.531518 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:22.531448    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:48:23.086307 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:23.086274    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f" event={"ID":"f64bc0a9-db3a-4e00-90db-4d81098629b2","Type":"ContainerStarted","Data":"c39bb2b6f3dc815760e5bc7962253cfc75dee5d48533d4891ad86d3b9c597c8a"}
Sep 16 18:48:23.086497 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:23.086328    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f" event={"ID":"f64bc0a9-db3a-4e00-90db-4d81098629b2","Type":"ContainerStarted","Data":"c5e8f0ee9777e24752f243b04319082d9b25068c9ba28f43fc2410de60e63333"}
Sep 16 18:48:23.111232 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:23.111182    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f" podStartSLOduration=2.111166067 podStartE2EDuration="2.111166067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:23.110386125 +0000 UTC m=+1940.975290174" watchObservedRunningTime="2026-09-16 18:48:23.111166067 +0000 UTC m=+1940.976070115"
Sep 16 18:48:23.243015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:23.242980    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:23.249115 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:23.249085    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:24.092695 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:24.092659    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:24.093762 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:24.093742    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:24.379786 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:24.379709    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:25.032024 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:25.031992    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:25.126332 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:25.126304    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:25.225319 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:25.225280    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:26.110944 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:26.110915    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:26.884278 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:26.884246    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:27.853479 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:27.853447    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:28.609144 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:28.609099    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:28.711093 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:28.711043    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:28.718210 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:28.718175    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:48:29.246660 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:29.246630    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:29.392644 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:29.392615    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:34.952536 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:34.952495    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:35.095297 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:35.095254    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj"]
Sep 16 18:48:35.099181 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:35.099158    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj"
Sep 16 18:48:35.102574 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:35.102552    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:48:35.102863 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:35.102843    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:48:35.110918 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:35.110894    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj"]
Sep 16 18:48:35.207605 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:35.207517    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/056b1280-f941-4ef7-bf0f-9c05d0aaaa68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj\" (UID: \"056b1280-f941-4ef7-bf0f-9c05d0aaaa68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj"
Sep 16 18:48:35.246597 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:35.246564    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:35.308279 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:35.308244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/056b1280-f941-4ef7-bf0f-9c05d0aaaa68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj\" (UID: \"056b1280-f941-4ef7-bf0f-9c05d0aaaa68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj"
Sep 16 18:48:35.309195 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:35.309171    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/056b1280-f941-4ef7-bf0f-9c05d0aaaa68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj\" (UID: \"056b1280-f941-4ef7-bf0f-9c05d0aaaa68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj"
Sep 16 18:48:35.411069 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:35.411029    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj"
Sep 16 18:48:35.690680 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:48:35.690643    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056b1280_f941_4ef7_bf0f_9c05d0aaaa68.slice/crio-f664c197c71e8b16191efc46e7eba3acaf4e8e5b78b7d53cbe66b5ff0810feea WatchSource:0}: Error finding container f664c197c71e8b16191efc46e7eba3acaf4e8e5b78b7d53cbe66b5ff0810feea: Status 404 returned error can't find the container with id f664c197c71e8b16191efc46e7eba3acaf4e8e5b78b7d53cbe66b5ff0810feea
Sep 16 18:48:35.690956 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:35.690930    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj"]
Sep 16 18:48:36.167353 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:36.167254    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj" event={"ID":"056b1280-f941-4ef7-bf0f-9c05d0aaaa68","Type":"ContainerStarted","Data":"c408fae62a6427f2ea6c6c4336bebd51884ae8d3689e4402e45d8a3f966739e3"}
Sep 16 18:48:36.167353 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:36.167316    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj" event={"ID":"056b1280-f941-4ef7-bf0f-9c05d0aaaa68","Type":"ContainerStarted","Data":"f664c197c71e8b16191efc46e7eba3acaf4e8e5b78b7d53cbe66b5ff0810feea"}
Sep 16 18:48:36.186834 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:36.186775    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj" podStartSLOduration=1.186758531 podStartE2EDuration="1.186758531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:36.184435778 +0000 UTC m=+1954.049339827" watchObservedRunningTime="2026-09-16 18:48:36.186758531 +0000 UTC m=+1954.051662578"
Sep 16 18:48:36.361648 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:36.361350    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:37.175421 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:37.175380    2562 generic.go:356] "Generic (PLEG): container finished" podID="056b1280-f941-4ef7-bf0f-9c05d0aaaa68" containerID="c408fae62a6427f2ea6c6c4336bebd51884ae8d3689e4402e45d8a3f966739e3" exitCode=0
Sep 16 18:48:37.176017 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:37.175470    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj" event={"ID":"056b1280-f941-4ef7-bf0f-9c05d0aaaa68","Type":"ContainerDied","Data":"c408fae62a6427f2ea6c6c4336bebd51884ae8d3689e4402e45d8a3f966739e3"}
Sep 16 18:48:37.376249 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:37.376210    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:38.253399 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:38.253363    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:38.323161 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:38.323132    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj"
Sep 16 18:48:38.449003 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:38.448525    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/056b1280-f941-4ef7-bf0f-9c05d0aaaa68-maas-api-service-ca\") pod \"056b1280-f941-4ef7-bf0f-9c05d0aaaa68\" (UID: \"056b1280-f941-4ef7-bf0f-9c05d0aaaa68\") "
Sep 16 18:48:38.449003 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:38.448874    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056b1280-f941-4ef7-bf0f-9c05d0aaaa68-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "056b1280-f941-4ef7-bf0f-9c05d0aaaa68" (UID: "056b1280-f941-4ef7-bf0f-9c05d0aaaa68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:38.549727 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:38.549662    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/056b1280-f941-4ef7-bf0f-9c05d0aaaa68-maas-api-service-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:39.191804 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:39.191766    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj"
Sep 16 18:48:39.191967 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:39.191763    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj" event={"ID":"056b1280-f941-4ef7-bf0f-9c05d0aaaa68","Type":"ContainerDied","Data":"f664c197c71e8b16191efc46e7eba3acaf4e8e5b78b7d53cbe66b5ff0810feea"}
Sep 16 18:48:39.191967 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:39.191898    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f664c197c71e8b16191efc46e7eba3acaf4e8e5b78b7d53cbe66b5ff0810feea"
Sep 16 18:48:39.872875 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:39.872830    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:39.928306 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:39.928270    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"]
Sep 16 18:48:39.937691 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:39.937655    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"]
Sep 16 18:48:39.944521 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:39.944457    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"]
Sep 16 18:48:39.977877 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:39.977838    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"]
Sep 16 18:48:40.088132 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:40.088077    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:40.181802 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:40.181719    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:40.942056 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:40.942019    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:41.845531 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:41.845497    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:42.586360 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:42.586326    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:43.431560 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:43.431521    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:44.159317 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:44.159263    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:44.861752 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:44.861714    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:45.126506 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:45.126394    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:47.248323 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.248292    2562 generic.go:356] "Generic (PLEG): container finished" podID="f64bc0a9-db3a-4e00-90db-4d81098629b2" containerID="c39bb2b6f3dc815760e5bc7962253cfc75dee5d48533d4891ad86d3b9c597c8a" exitCode=0
Sep 16 18:48:47.248799 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.248363    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f" event={"ID":"f64bc0a9-db3a-4e00-90db-4d81098629b2","Type":"ContainerDied","Data":"c39bb2b6f3dc815760e5bc7962253cfc75dee5d48533d4891ad86d3b9c597c8a"}
Sep 16 18:48:47.248853 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.248832    2562 scope.go:117] "RemoveContainer" containerID="c39bb2b6f3dc815760e5bc7962253cfc75dee5d48533d4891ad86d3b9c597c8a"
Sep 16 18:48:47.250057 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.250028    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:48:47.250130 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.250096    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:48:47.250170 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.250130    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:48:47.780955 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.780917    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t"]
Sep 16 18:48:47.781327 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.781313    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056b1280-f941-4ef7-bf0f-9c05d0aaaa68" containerName="revoke-keys"
Sep 16 18:48:47.781386 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.781330    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="056b1280-f941-4ef7-bf0f-9c05d0aaaa68" containerName="revoke-keys"
Sep 16 18:48:47.781428 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.781419    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="056b1280-f941-4ef7-bf0f-9c05d0aaaa68" containerName="revoke-keys"
Sep 16 18:48:47.784606 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.784586    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t"
Sep 16 18:48:47.787839 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.787818    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:48:47.799295 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.799268    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:48:47.810019 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.809968    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t"]
Sep 16 18:48:47.866567 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.863228    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51121220-40b0-4e10-8976-5e9cd3b01de0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t\" (UID: \"51121220-40b0-4e10-8976-5e9cd3b01de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t"
Sep 16 18:48:47.970501 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.970325    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:47.971401 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.971368    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51121220-40b0-4e10-8976-5e9cd3b01de0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t\" (UID: \"51121220-40b0-4e10-8976-5e9cd3b01de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t"
Sep 16 18:48:47.972451 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:47.972308    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51121220-40b0-4e10-8976-5e9cd3b01de0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t\" (UID: \"51121220-40b0-4e10-8976-5e9cd3b01de0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t"
Sep 16 18:48:48.096435 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:48.096353    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t"
Sep 16 18:48:48.262561 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:48.262495    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f" event={"ID":"f64bc0a9-db3a-4e00-90db-4d81098629b2","Type":"ContainerStarted","Data":"e623c0b2d09800e1746b996a38efb5377c4210c74b1324e063a4e237b86ef230"}
Sep 16 18:48:48.399011 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:48.398976    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t"]
Sep 16 18:48:48.399155 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:48:48.399139    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51121220_40b0_4e10_8976_5e9cd3b01de0.slice/crio-e4f563bdc6686020bc993f6c42f80bf1f7402badcc185036e204e8f02890c421 WatchSource:0}: Error finding container e4f563bdc6686020bc993f6c42f80bf1f7402badcc185036e204e8f02890c421: Status 404 returned error can't find the container with id e4f563bdc6686020bc993f6c42f80bf1f7402badcc185036e204e8f02890c421
Sep 16 18:48:49.082678 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:49.082637    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:49.243054 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:49.243013    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:49.244053 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:49.244022    2562 patch_prober.go:28] interesting pod/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 16 18:48:49.244180 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:49.244076    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f" podUID="f64bc0a9-db3a-4e00-90db-4d81098629b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 16 18:48:49.278480 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:49.278427    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t" event={"ID":"51121220-40b0-4e10-8976-5e9cd3b01de0","Type":"ContainerStarted","Data":"4a1776c5a293786f638534f8e258fd3a28002fc6fd918420c1c323393de79b96"}
Sep 16 18:48:49.279011 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:49.278501    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t" event={"ID":"51121220-40b0-4e10-8976-5e9cd3b01de0","Type":"ContainerStarted","Data":"e4f563bdc6686020bc993f6c42f80bf1f7402badcc185036e204e8f02890c421"}
Sep 16 18:48:49.296985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:49.296922    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t" podStartSLOduration=2.296905517 podStartE2EDuration="2.296905517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:49.295001705 +0000 UTC m=+1967.159905752" watchObservedRunningTime="2026-09-16 18:48:49.296905517 +0000 UTC m=+1967.161809565"
Sep 16 18:48:49.887799 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:49.887762    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:50.242857 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:50.242822    2562 patch_prober.go:28] interesting pod/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 16 18:48:50.243010 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:50.242874    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f" podUID="f64bc0a9-db3a-4e00-90db-4d81098629b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 16 18:48:50.289741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:50.289703    2562 generic.go:356] "Generic (PLEG): container finished" podID="51121220-40b0-4e10-8976-5e9cd3b01de0" containerID="4a1776c5a293786f638534f8e258fd3a28002fc6fd918420c1c323393de79b96" exitCode=0
Sep 16 18:48:50.290125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:50.289746    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t" event={"ID":"51121220-40b0-4e10-8976-5e9cd3b01de0","Type":"ContainerDied","Data":"4a1776c5a293786f638534f8e258fd3a28002fc6fd918420c1c323393de79b96"}
Sep 16 18:48:50.512927 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:50.512826    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d8a600-29826405-dtmpq"]
Sep 16 18:48:50.517324 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:50.517292    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d8a600-29826405-dtmpq"]
Sep 16 18:48:50.529461 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:50.529428    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:50.590568 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:50.590166    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d98ec06-4e04-4e74-ab0b-fe753b16ac64" path="/var/lib/kubelet/pods/6d98ec06-4e04-4e74-ab0b-fe753b16ac64/volumes"
Sep 16 18:48:50.634034 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:50.633994    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:50.743611 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:50.743576    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:50.852847 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:50.852750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:50.885722 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:50.885642    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj"]
Sep 16 18:48:50.887626 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:50.887593    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d8a600-hcmlj"]
Sep 16 18:48:51.024720 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:51.024682    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:51.243698 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:51.243663    2562 patch_prober.go:28] interesting pod/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 16 18:48:51.243854 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:51.243720    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f" podUID="f64bc0a9-db3a-4e00-90db-4d81098629b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 16 18:48:51.273480 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:51.273440    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:51.479819 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:51.479787    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t"
Sep 16 18:48:51.610269 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:51.610162    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51121220-40b0-4e10-8976-5e9cd3b01de0-maas-api-service-ca\") pod \"51121220-40b0-4e10-8976-5e9cd3b01de0\" (UID: \"51121220-40b0-4e10-8976-5e9cd3b01de0\") "
Sep 16 18:48:51.610586 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:51.610532    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51121220-40b0-4e10-8976-5e9cd3b01de0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51121220-40b0-4e10-8976-5e9cd3b01de0" (UID: "51121220-40b0-4e10-8976-5e9cd3b01de0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:51.711141 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:51.711104    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51121220-40b0-4e10-8976-5e9cd3b01de0-maas-api-service-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:51.978870 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:51.978831    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:52.243015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:52.242918    2562 patch_prober.go:28] interesting pod/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 16 18:48:52.243015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:52.242992    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f" podUID="f64bc0a9-db3a-4e00-90db-4d81098629b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 16 18:48:52.304306 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:52.304270    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t"
Sep 16 18:48:52.304489 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:52.304304    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t" event={"ID":"51121220-40b0-4e10-8976-5e9cd3b01de0","Type":"ContainerDied","Data":"e4f563bdc6686020bc993f6c42f80bf1f7402badcc185036e204e8f02890c421"}
Sep 16 18:48:52.304489 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:52.304361    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4f563bdc6686020bc993f6c42f80bf1f7402badcc185036e204e8f02890c421"
Sep 16 18:48:52.444810 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:52.444772    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:48:52.553059 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:52.552972    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="056b1280-f941-4ef7-bf0f-9c05d0aaaa68" path="/var/lib/kubelet/pods/056b1280-f941-4ef7-bf0f-9c05d0aaaa68/volumes"
Sep 16 18:48:52.814678 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:52.814583    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:53.243059 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:53.243023    2562 patch_prober.go:28] interesting pod/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 16 18:48:53.243240 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:53.243082    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f" podUID="f64bc0a9-db3a-4e00-90db-4d81098629b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 16 18:48:53.629320 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:53.629208    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:54.092861 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:54.092824    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:54.247613 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:54.247576    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:54.319391 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:54.319363    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:48:54.594355 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:54.594319    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:54.948473 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:54.948391    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj" podUID="b876aa39-20ce-49c3-9d52-684d4590bede" containerName="main" containerID="cri-o://405a9874ab696f40c124ddd648e4916771de7213e5f08c142be27b16a8689512" gracePeriod=45
Sep 16 18:48:54.989194 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:54.989150    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d" podUID="60248414-4995-4f51-b233-5dd9f38968a8" containerName="main" containerID="cri-o://8ea9b950df8b7e6b48426a9f8316bfba50d528764499ea778a87ed1243de73ff" gracePeriod=45
Sep 16 18:48:54.993218 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:54.993184    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5" podUID="8ef68f3a-a4c3-4b67-9d44-2de265669783" containerName="main" containerID="cri-o://de1d5422e29e83823f8081fc51ed6f1892260914d2e48d3a9e41fc350a76433d" gracePeriod=45
Sep 16 18:48:55.000640 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.000602    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd" podUID="ee7d27e9-1bbf-429f-b3b2-052f8e704fb3" containerName="main" containerID="cri-o://728505d1be4c8a519cdae425e2f47cba5db1a5ed17cb4732f43f7d68993e7618" gracePeriod=45
Sep 16 18:48:55.303138 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.303110    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:48:55.328941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.328906    2562 generic.go:356] "Generic (PLEG): container finished" podID="60248414-4995-4f51-b233-5dd9f38968a8" containerID="8ea9b950df8b7e6b48426a9f8316bfba50d528764499ea778a87ed1243de73ff" exitCode=0
Sep 16 18:48:55.329080 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.329007    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d" event={"ID":"60248414-4995-4f51-b233-5dd9f38968a8","Type":"ContainerDied","Data":"8ea9b950df8b7e6b48426a9f8316bfba50d528764499ea778a87ed1243de73ff"}
Sep 16 18:48:55.329080 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.329043    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d" event={"ID":"60248414-4995-4f51-b233-5dd9f38968a8","Type":"ContainerDied","Data":"76677612dceccd295c7c4901cd222a0d36a00064288a08b7fa5bb0a1d6046f02"}
Sep 16 18:48:55.329080 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.329061    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76677612dceccd295c7c4901cd222a0d36a00064288a08b7fa5bb0a1d6046f02"
Sep 16 18:48:55.331953 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.331919    2562 generic.go:356] "Generic (PLEG): container finished" podID="b876aa39-20ce-49c3-9d52-684d4590bede" containerID="405a9874ab696f40c124ddd648e4916771de7213e5f08c142be27b16a8689512" exitCode=0
Sep 16 18:48:55.332135 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.331974    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"
Sep 16 18:48:55.332135 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.332003    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj" event={"ID":"b876aa39-20ce-49c3-9d52-684d4590bede","Type":"ContainerDied","Data":"405a9874ab696f40c124ddd648e4916771de7213e5f08c142be27b16a8689512"}
Sep 16 18:48:55.332135 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.332038    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj" event={"ID":"b876aa39-20ce-49c3-9d52-684d4590bede","Type":"ContainerDied","Data":"32c07c1293f46731fe8ce90e3446d744c9bf5f800387141c6168fad6713cd1d7"}
Sep 16 18:48:55.332135 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.332054    2562 scope.go:117] "RemoveContainer" containerID="405a9874ab696f40c124ddd648e4916771de7213e5f08c142be27b16a8689512"
Sep 16 18:48:55.334003 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.333980    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:48:55.334135 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.334050    2562 generic.go:356] "Generic (PLEG): container finished" podID="ee7d27e9-1bbf-429f-b3b2-052f8e704fb3" containerID="728505d1be4c8a519cdae425e2f47cba5db1a5ed17cb4732f43f7d68993e7618" exitCode=0
Sep 16 18:48:55.334135 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.334072    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd" event={"ID":"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3","Type":"ContainerDied","Data":"728505d1be4c8a519cdae425e2f47cba5db1a5ed17cb4732f43f7d68993e7618"}
Sep 16 18:48:55.336261 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.336011    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ef68f3a-a4c3-4b67-9d44-2de265669783" containerID="de1d5422e29e83823f8081fc51ed6f1892260914d2e48d3a9e41fc350a76433d" exitCode=0
Sep 16 18:48:55.336261 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.336207    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5" event={"ID":"8ef68f3a-a4c3-4b67-9d44-2de265669783","Type":"ContainerDied","Data":"de1d5422e29e83823f8081fc51ed6f1892260914d2e48d3a9e41fc350a76433d"}
Sep 16 18:48:55.336261 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.336219    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"
Sep 16 18:48:55.336261 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.336242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5" event={"ID":"8ef68f3a-a4c3-4b67-9d44-2de265669783","Type":"ContainerDied","Data":"22d1b4e97ce01916d3b26bd907640b3d5ff703d5ef66e161bd77abecc63e4d14"}
Sep 16 18:48:55.339121 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.339101    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:48:55.343437 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.343421    2562 scope.go:117] "RemoveContainer" containerID="405a9874ab696f40c124ddd648e4916771de7213e5f08c142be27b16a8689512"
Sep 16 18:48:55.343760 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:48:55.343736    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405a9874ab696f40c124ddd648e4916771de7213e5f08c142be27b16a8689512\": container with ID starting with 405a9874ab696f40c124ddd648e4916771de7213e5f08c142be27b16a8689512 not found: ID does not exist" containerID="405a9874ab696f40c124ddd648e4916771de7213e5f08c142be27b16a8689512"
Sep 16 18:48:55.343827 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.343772    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405a9874ab696f40c124ddd648e4916771de7213e5f08c142be27b16a8689512"} err="failed to get container status \"405a9874ab696f40c124ddd648e4916771de7213e5f08c142be27b16a8689512\": rpc error: code = NotFound desc = could not find container \"405a9874ab696f40c124ddd648e4916771de7213e5f08c142be27b16a8689512\": container with ID starting with 405a9874ab696f40c124ddd648e4916771de7213e5f08c142be27b16a8689512 not found: ID does not exist"
Sep 16 18:48:55.343827 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.343795    2562 scope.go:117] "RemoveContainer" containerID="de1d5422e29e83823f8081fc51ed6f1892260914d2e48d3a9e41fc350a76433d"
Sep 16 18:48:55.344062 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.344043    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:48:55.346667 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.346646    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-tls-certs\") pod \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") "
Sep 16 18:48:55.346766 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.346679    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-home\") pod \"8ef68f3a-a4c3-4b67-9d44-2de265669783\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") "
Sep 16 18:48:55.346766 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.346714    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-dshm\") pod \"8ef68f3a-a4c3-4b67-9d44-2de265669783\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") "
Sep 16 18:48:55.346766 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.346746    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ef68f3a-a4c3-4b67-9d44-2de265669783-tls-certs\") pod \"8ef68f3a-a4c3-4b67-9d44-2de265669783\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") "
Sep 16 18:48:55.346929 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.346776    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8rxn\" (UniqueName: \"kubernetes.io/projected/60248414-4995-4f51-b233-5dd9f38968a8-kube-api-access-d8rxn\") pod \"60248414-4995-4f51-b233-5dd9f38968a8\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") "
Sep 16 18:48:55.346929 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.346803    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-model-cache\") pod \"b876aa39-20ce-49c3-9d52-684d4590bede\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") "
Sep 16 18:48:55.346929 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.346834    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-tmp-dir\") pod \"8ef68f3a-a4c3-4b67-9d44-2de265669783\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") "
Sep 16 18:48:55.346929 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.346861    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-model-cache\") pod \"8ef68f3a-a4c3-4b67-9d44-2de265669783\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") "
Sep 16 18:48:55.346929 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.346892    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-dshm\") pod \"60248414-4995-4f51-b233-5dd9f38968a8\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") "
Sep 16 18:48:55.346929 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.346917    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbbm9\" (UniqueName: \"kubernetes.io/projected/b876aa39-20ce-49c3-9d52-684d4590bede-kube-api-access-mbbm9\") pod \"b876aa39-20ce-49c3-9d52-684d4590bede\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") "
Sep 16 18:48:55.347252 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.346947    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-home\") pod \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") "
Sep 16 18:48:55.347252 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.346972    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-dshm\") pod \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") "
Sep 16 18:48:55.347252 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.347004    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-model-cache\") pod \"60248414-4995-4f51-b233-5dd9f38968a8\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") "
Sep 16 18:48:55.347252 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.347039    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ft77\" (UniqueName: \"kubernetes.io/projected/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-kube-api-access-8ft77\") pod \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") "
Sep 16 18:48:55.347252 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.347085    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td87x\" (UniqueName: \"kubernetes.io/projected/8ef68f3a-a4c3-4b67-9d44-2de265669783-kube-api-access-td87x\") pod \"8ef68f3a-a4c3-4b67-9d44-2de265669783\" (UID: \"8ef68f3a-a4c3-4b67-9d44-2de265669783\") "
Sep 16 18:48:55.347252 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.347114    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b876aa39-20ce-49c3-9d52-684d4590bede-tls-certs\") pod \"b876aa39-20ce-49c3-9d52-684d4590bede\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") "
Sep 16 18:48:55.347252 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.347141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-tmp-dir\") pod \"b876aa39-20ce-49c3-9d52-684d4590bede\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") "
Sep 16 18:48:55.347252 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.347164    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-tmp-dir\") pod \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") "
Sep 16 18:48:55.347252 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.347196    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-home\") pod \"60248414-4995-4f51-b233-5dd9f38968a8\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") "
Sep 16 18:48:55.347252 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.347249    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-tmp-dir\") pod \"60248414-4995-4f51-b233-5dd9f38968a8\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") "
Sep 16 18:48:55.350343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.347294    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-home\") pod \"b876aa39-20ce-49c3-9d52-684d4590bede\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") "
Sep 16 18:48:55.350343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.347332    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-model-cache\") pod \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\" (UID: \"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3\") "
Sep 16 18:48:55.350343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.347369    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-dshm\") pod \"b876aa39-20ce-49c3-9d52-684d4590bede\" (UID: \"b876aa39-20ce-49c3-9d52-684d4590bede\") "
Sep 16 18:48:55.350343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.347399    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60248414-4995-4f51-b233-5dd9f38968a8-tls-certs\") pod \"60248414-4995-4f51-b233-5dd9f38968a8\" (UID: \"60248414-4995-4f51-b233-5dd9f38968a8\") "
Sep 16 18:48:55.350343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.348174    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee7d27e9-1bbf-429f-b3b2-052f8e704fb3" (UID: "ee7d27e9-1bbf-429f-b3b2-052f8e704fb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.350343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.348600    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-home" (OuterVolumeSpecName: "home") pod "8ef68f3a-a4c3-4b67-9d44-2de265669783" (UID: "8ef68f3a-a4c3-4b67-9d44-2de265669783"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.350343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.348829    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-home" (OuterVolumeSpecName: "home") pod "b876aa39-20ce-49c3-9d52-684d4590bede" (UID: "b876aa39-20ce-49c3-9d52-684d4590bede"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.350343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.349020    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-home" (OuterVolumeSpecName: "home") pod "60248414-4995-4f51-b233-5dd9f38968a8" (UID: "60248414-4995-4f51-b233-5dd9f38968a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.350343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.349204    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee7d27e9-1bbf-429f-b3b2-052f8e704fb3" (UID: "ee7d27e9-1bbf-429f-b3b2-052f8e704fb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.350343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.349502    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60248414-4995-4f51-b233-5dd9f38968a8" (UID: "60248414-4995-4f51-b233-5dd9f38968a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.350343 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.349745    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b876aa39-20ce-49c3-9d52-684d4590bede" (UID: "b876aa39-20ce-49c3-9d52-684d4590bede"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.351442 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.351416    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-dshm" (OuterVolumeSpecName: "dshm") pod "8ef68f3a-a4c3-4b67-9d44-2de265669783" (UID: "8ef68f3a-a4c3-4b67-9d44-2de265669783"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.351664 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.351642    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ef68f3a-a4c3-4b67-9d44-2de265669783" (UID: "8ef68f3a-a4c3-4b67-9d44-2de265669783"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.351981 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.351957    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-model-cache" (OuterVolumeSpecName: "model-cache") pod "b876aa39-20ce-49c3-9d52-684d4590bede" (UID: "b876aa39-20ce-49c3-9d52-684d4590bede"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.354641 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.354184    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-dshm" (OuterVolumeSpecName: "dshm") pod "ee7d27e9-1bbf-429f-b3b2-052f8e704fb3" (UID: "ee7d27e9-1bbf-429f-b3b2-052f8e704fb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.354641 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.354372    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-home" (OuterVolumeSpecName: "home") pod "ee7d27e9-1bbf-429f-b3b2-052f8e704fb3" (UID: "ee7d27e9-1bbf-429f-b3b2-052f8e704fb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.356426 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.356205    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ef68f3a-a4c3-4b67-9d44-2de265669783" (UID: "8ef68f3a-a4c3-4b67-9d44-2de265669783"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.357515 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.356913    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60248414-4995-4f51-b233-5dd9f38968a8-kube-api-access-d8rxn" (OuterVolumeSpecName: "kube-api-access-d8rxn") pod "60248414-4995-4f51-b233-5dd9f38968a8" (UID: "60248414-4995-4f51-b233-5dd9f38968a8"). InnerVolumeSpecName "kube-api-access-d8rxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:55.357515 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.357146    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "60248414-4995-4f51-b233-5dd9f38968a8" (UID: "60248414-4995-4f51-b233-5dd9f38968a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.357515 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.357205    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60248414-4995-4f51-b233-5dd9f38968a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60248414-4995-4f51-b233-5dd9f38968a8" (UID: "60248414-4995-4f51-b233-5dd9f38968a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:55.359259 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.358342    2562 scope.go:117] "RemoveContainer" containerID="de1d5422e29e83823f8081fc51ed6f1892260914d2e48d3a9e41fc350a76433d"
Sep 16 18:48:55.359741 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:48:55.359601    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1d5422e29e83823f8081fc51ed6f1892260914d2e48d3a9e41fc350a76433d\": container with ID starting with de1d5422e29e83823f8081fc51ed6f1892260914d2e48d3a9e41fc350a76433d not found: ID does not exist" containerID="de1d5422e29e83823f8081fc51ed6f1892260914d2e48d3a9e41fc350a76433d"
Sep 16 18:48:55.359741 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.359639    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1d5422e29e83823f8081fc51ed6f1892260914d2e48d3a9e41fc350a76433d"} err="failed to get container status \"de1d5422e29e83823f8081fc51ed6f1892260914d2e48d3a9e41fc350a76433d\": rpc error: code = NotFound desc = could not find container \"de1d5422e29e83823f8081fc51ed6f1892260914d2e48d3a9e41fc350a76433d\": container with ID starting with de1d5422e29e83823f8081fc51ed6f1892260914d2e48d3a9e41fc350a76433d not found: ID does not exist"
Sep 16 18:48:55.360867 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.360821    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef68f3a-a4c3-4b67-9d44-2de265669783-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ef68f3a-a4c3-4b67-9d44-2de265669783" (UID: "8ef68f3a-a4c3-4b67-9d44-2de265669783"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:55.360867 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.360822    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-dshm" (OuterVolumeSpecName: "dshm") pod "b876aa39-20ce-49c3-9d52-684d4590bede" (UID: "b876aa39-20ce-49c3-9d52-684d4590bede"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.361164 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.361128    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef68f3a-a4c3-4b67-9d44-2de265669783-kube-api-access-td87x" (OuterVolumeSpecName: "kube-api-access-td87x") pod "8ef68f3a-a4c3-4b67-9d44-2de265669783" (UID: "8ef68f3a-a4c3-4b67-9d44-2de265669783"). InnerVolumeSpecName "kube-api-access-td87x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:55.361412 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.361369    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee7d27e9-1bbf-429f-b3b2-052f8e704fb3" (UID: "ee7d27e9-1bbf-429f-b3b2-052f8e704fb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:55.363605 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.363302    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-kube-api-access-8ft77" (OuterVolumeSpecName: "kube-api-access-8ft77") pod "ee7d27e9-1bbf-429f-b3b2-052f8e704fb3" (UID: "ee7d27e9-1bbf-429f-b3b2-052f8e704fb3"). InnerVolumeSpecName "kube-api-access-8ft77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:55.364850 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.364817    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-dshm" (OuterVolumeSpecName: "dshm") pod "60248414-4995-4f51-b233-5dd9f38968a8" (UID: "60248414-4995-4f51-b233-5dd9f38968a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:55.365163 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.365129    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b876aa39-20ce-49c3-9d52-684d4590bede-kube-api-access-mbbm9" (OuterVolumeSpecName: "kube-api-access-mbbm9") pod "b876aa39-20ce-49c3-9d52-684d4590bede" (UID: "b876aa39-20ce-49c3-9d52-684d4590bede"). InnerVolumeSpecName "kube-api-access-mbbm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:55.365525 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.365498    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b876aa39-20ce-49c3-9d52-684d4590bede-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b876aa39-20ce-49c3-9d52-684d4590bede" (UID: "b876aa39-20ce-49c3-9d52-684d4590bede"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:55.424743 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.424699    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:55.448919 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.448884    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.448919 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.448920    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.448934    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.448945    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.448960    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbbm9\" (UniqueName: \"kubernetes.io/projected/b876aa39-20ce-49c3-9d52-684d4590bede-kube-api-access-mbbm9\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.448977    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.448989    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449000    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449014    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ft77\" (UniqueName: \"kubernetes.io/projected/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-kube-api-access-8ft77\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449026    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td87x\" (UniqueName: \"kubernetes.io/projected/8ef68f3a-a4c3-4b67-9d44-2de265669783-kube-api-access-td87x\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449038    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b876aa39-20ce-49c3-9d52-684d4590bede-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449050    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449060    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449069    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449081    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60248414-4995-4f51-b233-5dd9f38968a8-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449093    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449105    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449115    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b876aa39-20ce-49c3-9d52-684d4590bede-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449127    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60248414-4995-4f51-b233-5dd9f38968a8-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449768 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449140    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449768 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449152    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449768 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449163    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ef68f3a-a4c3-4b67-9d44-2de265669783-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449768 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449175    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ef68f3a-a4c3-4b67-9d44-2de265669783-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.449768 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.449187    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8rxn\" (UniqueName: \"kubernetes.io/projected/60248414-4995-4f51-b233-5dd9f38968a8-kube-api-access-d8rxn\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:48:55.659731 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.659670    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"]
Sep 16 18:48:55.663030 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.663004    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/distinct-2-w4-d8a600-kserve-755487895-hckwj"]
Sep 16 18:48:55.675532 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.675506    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"]
Sep 16 18:48:55.678435 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:55.678411    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/premium-w4-d8a600-kserve-6dd69dbbfc-9w4k5"]
Sep 16 18:48:56.345369 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:56.345266    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"
Sep 16 18:48:56.345369 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:56.345301    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd" event={"ID":"ee7d27e9-1bbf-429f-b3b2-052f8e704fb3","Type":"ContainerDied","Data":"dbc88887cc8e75db3852df767eeffce9d1da631b1881a1416e838e7ace458065"}
Sep 16 18:48:56.345911 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:56.345368    2562 scope.go:117] "RemoveContainer" containerID="728505d1be4c8a519cdae425e2f47cba5db1a5ed17cb4732f43f7d68993e7618"
Sep 16 18:48:56.346963 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:56.346613    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"
Sep 16 18:48:56.374445 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:56.374419    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"]
Sep 16 18:48:56.378314 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:56.378275    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/simulator-w4-d8a600-kserve-67cfc9f79b-lgngd"]
Sep 16 18:48:56.390622 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:56.390590    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"]
Sep 16 18:48:56.397153 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:56.397114    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d8a600/distinct-w4-d8a600-kserve-5684b96bc6-dc76d"]
Sep 16 18:48:56.552229 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:56.552190    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60248414-4995-4f51-b233-5dd9f38968a8" path="/var/lib/kubelet/pods/60248414-4995-4f51-b233-5dd9f38968a8/volumes"
Sep 16 18:48:56.552819 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:56.552798    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ef68f3a-a4c3-4b67-9d44-2de265669783" path="/var/lib/kubelet/pods/8ef68f3a-a4c3-4b67-9d44-2de265669783/volumes"
Sep 16 18:48:56.553369 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:56.553348    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b876aa39-20ce-49c3-9d52-684d4590bede" path="/var/lib/kubelet/pods/b876aa39-20ce-49c3-9d52-684d4590bede/volumes"
Sep 16 18:48:56.554296 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:56.554276    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee7d27e9-1bbf-429f-b3b2-052f8e704fb3" path="/var/lib/kubelet/pods/ee7d27e9-1bbf-429f-b3b2-052f8e704fb3/volumes"
Sep 16 18:48:57.566844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:57.566801    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:57.669394 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:57.669341    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:57.952572 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:57.952512    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"]
Sep 16 18:48:57.952785 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:57.952762    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh" podUID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerName="istio-proxy" containerID="cri-o://984caac5dd48ed418cfe92b012102fec01607fc7a0777c5e0b6394baaeba5dd9" gracePeriod=30
Sep 16 18:48:57.974838 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:57.974802    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:58.117799 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:48:58.117765    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:03.217261 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.217226    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:49:03.322444 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.322339    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec6d40bd-176d-4785-833d-d894891fbe1f-istiod-ca-cert\") pod \"ec6d40bd-176d-4785-833d-d894891fbe1f\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") "
Sep 16 18:49:03.322444 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.322399    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-workload-certs\") pod \"ec6d40bd-176d-4785-833d-d894891fbe1f\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") "
Sep 16 18:49:03.322734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.322447    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-workload-socket\") pod \"ec6d40bd-176d-4785-833d-d894891fbe1f\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") "
Sep 16 18:49:03.322734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.322493    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-credential-socket\") pod \"ec6d40bd-176d-4785-833d-d894891fbe1f\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") "
Sep 16 18:49:03.322734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.322529    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-envoy\") pod \"ec6d40bd-176d-4785-833d-d894891fbe1f\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") "
Sep 16 18:49:03.322734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.322597    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-token\") pod \"ec6d40bd-176d-4785-833d-d894891fbe1f\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") "
Sep 16 18:49:03.322734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.322633    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-data\") pod \"ec6d40bd-176d-4785-833d-d894891fbe1f\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") "
Sep 16 18:49:03.322734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.322680    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-podinfo\") pod \"ec6d40bd-176d-4785-833d-d894891fbe1f\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") "
Sep 16 18:49:03.322734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.322713    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rdzd\" (UniqueName: \"kubernetes.io/projected/ec6d40bd-176d-4785-833d-d894891fbe1f-kube-api-access-2rdzd\") pod \"ec6d40bd-176d-4785-833d-d894891fbe1f\" (UID: \"ec6d40bd-176d-4785-833d-d894891fbe1f\") "
Sep 16 18:49:03.323028 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.322747    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec6d40bd-176d-4785-833d-d894891fbe1f" (UID: "ec6d40bd-176d-4785-833d-d894891fbe1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:03.323028 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.322792    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec6d40bd-176d-4785-833d-d894891fbe1f" (UID: "ec6d40bd-176d-4785-833d-d894891fbe1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:03.323028 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.322813    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec6d40bd-176d-4785-833d-d894891fbe1f" (UID: "ec6d40bd-176d-4785-833d-d894891fbe1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:03.323028 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.322848    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6d40bd-176d-4785-833d-d894891fbe1f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec6d40bd-176d-4785-833d-d894891fbe1f" (UID: "ec6d40bd-176d-4785-833d-d894891fbe1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:03.323028 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.322905    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec6d40bd-176d-4785-833d-d894891fbe1f" (UID: "ec6d40bd-176d-4785-833d-d894891fbe1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:03.323263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.323065    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec6d40bd-176d-4785-833d-d894891fbe1f-istiod-ca-cert\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.323263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.323084    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-workload-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.323263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.323098    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-workload-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.323263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.323112    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-credential-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.323263 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.323127    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-data\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.324974 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.324946    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6d40bd-176d-4785-833d-d894891fbe1f-kube-api-access-2rdzd" (OuterVolumeSpecName: "kube-api-access-2rdzd") pod "ec6d40bd-176d-4785-833d-d894891fbe1f" (UID: "ec6d40bd-176d-4785-833d-d894891fbe1f"). InnerVolumeSpecName "kube-api-access-2rdzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:03.325098 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.324974    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec6d40bd-176d-4785-833d-d894891fbe1f" (UID: "ec6d40bd-176d-4785-833d-d894891fbe1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:03.325098 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.324985    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec6d40bd-176d-4785-833d-d894891fbe1f" (UID: "ec6d40bd-176d-4785-833d-d894891fbe1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:49:03.325098 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.325016    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec6d40bd-176d-4785-833d-d894891fbe1f" (UID: "ec6d40bd-176d-4785-833d-d894891fbe1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:03.397862 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.397823    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerID="984caac5dd48ed418cfe92b012102fec01607fc7a0777c5e0b6394baaeba5dd9" exitCode=0
Sep 16 18:49:03.398058 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.397885    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"
Sep 16 18:49:03.398058 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.397902    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh" event={"ID":"ec6d40bd-176d-4785-833d-d894891fbe1f","Type":"ContainerDied","Data":"984caac5dd48ed418cfe92b012102fec01607fc7a0777c5e0b6394baaeba5dd9"}
Sep 16 18:49:03.398058 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.397958    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh" event={"ID":"ec6d40bd-176d-4785-833d-d894891fbe1f","Type":"ContainerDied","Data":"9a5a559d7fe9cecbe7154ee82ceb09c15c9d58f755f05e5441d16958dce2681e"}
Sep 16 18:49:03.398058 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.397981    2562 scope.go:117] "RemoveContainer" containerID="984caac5dd48ed418cfe92b012102fec01607fc7a0777c5e0b6394baaeba5dd9"
Sep 16 18:49:03.409723 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.409699    2562 scope.go:117] "RemoveContainer" containerID="33f49ca09ec6ea51a537233df1753b70ec6baf0ef9ac1307545bed63bf7eff89"
Sep 16 18:49:03.419511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.419485    2562 scope.go:117] "RemoveContainer" containerID="984caac5dd48ed418cfe92b012102fec01607fc7a0777c5e0b6394baaeba5dd9"
Sep 16 18:49:03.419976 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:49:03.419947    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"984caac5dd48ed418cfe92b012102fec01607fc7a0777c5e0b6394baaeba5dd9\": container with ID starting with 984caac5dd48ed418cfe92b012102fec01607fc7a0777c5e0b6394baaeba5dd9 not found: ID does not exist" containerID="984caac5dd48ed418cfe92b012102fec01607fc7a0777c5e0b6394baaeba5dd9"
Sep 16 18:49:03.420079 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.419986    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"984caac5dd48ed418cfe92b012102fec01607fc7a0777c5e0b6394baaeba5dd9"} err="failed to get container status \"984caac5dd48ed418cfe92b012102fec01607fc7a0777c5e0b6394baaeba5dd9\": rpc error: code = NotFound desc = could not find container \"984caac5dd48ed418cfe92b012102fec01607fc7a0777c5e0b6394baaeba5dd9\": container with ID starting with 984caac5dd48ed418cfe92b012102fec01607fc7a0777c5e0b6394baaeba5dd9 not found: ID does not exist"
Sep 16 18:49:03.420079 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.420013    2562 scope.go:117] "RemoveContainer" containerID="33f49ca09ec6ea51a537233df1753b70ec6baf0ef9ac1307545bed63bf7eff89"
Sep 16 18:49:03.420359 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:49:03.420342    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f49ca09ec6ea51a537233df1753b70ec6baf0ef9ac1307545bed63bf7eff89\": container with ID starting with 33f49ca09ec6ea51a537233df1753b70ec6baf0ef9ac1307545bed63bf7eff89 not found: ID does not exist" containerID="33f49ca09ec6ea51a537233df1753b70ec6baf0ef9ac1307545bed63bf7eff89"
Sep 16 18:49:03.420403 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.420364    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f49ca09ec6ea51a537233df1753b70ec6baf0ef9ac1307545bed63bf7eff89"} err="failed to get container status \"33f49ca09ec6ea51a537233df1753b70ec6baf0ef9ac1307545bed63bf7eff89\": rpc error: code = NotFound desc = could not find container \"33f49ca09ec6ea51a537233df1753b70ec6baf0ef9ac1307545bed63bf7eff89\": container with ID starting with 33f49ca09ec6ea51a537233df1753b70ec6baf0ef9ac1307545bed63bf7eff89 not found: ID does not exist"
Sep 16 18:49:03.423932 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.423903    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-envoy\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.423932 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.423934    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-token\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.424046 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.423943    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec6d40bd-176d-4785-833d-d894891fbe1f-istio-podinfo\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.424046 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.423952    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rdzd\" (UniqueName: \"kubernetes.io/projected/ec6d40bd-176d-4785-833d-d894891fbe1f-kube-api-access-2rdzd\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.458359 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.458309    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"]
Sep 16 18:49:03.465720 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.465681    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d8a600-openshift-default-9d45d7c78-d7hrh"]
Sep 16 18:49:03.787621 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.787528    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:03.901214 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.901181    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:03.994216 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:03.994171    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:04.106203 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:04.106119    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:04.119279 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:04.119245    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t"]
Sep 16 18:49:04.121593 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:04.121564    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1f16eb-6cd6t"]
Sep 16 18:49:04.202380 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:04.202334    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:04.483996 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:04.483959    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:04.550902 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:04.550866    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51121220-40b0-4e10-8976-5e9cd3b01de0" path="/var/lib/kubelet/pods/51121220-40b0-4e10-8976-5e9cd3b01de0/volumes"
Sep 16 18:49:04.551520 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:04.551497    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec6d40bd-176d-4785-833d-d894891fbe1f" path="/var/lib/kubelet/pods/ec6d40bd-176d-4785-833d-d894891fbe1f/volumes"
Sep 16 18:49:04.667843 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:04.667811    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:05.484684 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:05.484653    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:06.435591 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:06.435534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:07.184282 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:07.184239    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:09.515753 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:09.515707    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:49:11.057879 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.057840    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:11.160098 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.160061    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:11.326052 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.325958    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:11.698275 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698193    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5"]
Sep 16 18:49:11.698613 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698595    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51121220-40b0-4e10-8976-5e9cd3b01de0" containerName="revoke-keys"
Sep 16 18:49:11.698657 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698613    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="51121220-40b0-4e10-8976-5e9cd3b01de0" containerName="revoke-keys"
Sep 16 18:49:11.698657 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698624    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerName="istio-proxy"
Sep 16 18:49:11.698657 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698629    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerName="istio-proxy"
Sep 16 18:49:11.698657 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698637    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee7d27e9-1bbf-429f-b3b2-052f8e704fb3" containerName="main"
Sep 16 18:49:11.698657 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698643    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee7d27e9-1bbf-429f-b3b2-052f8e704fb3" containerName="main"
Sep 16 18:49:11.698657 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698652    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b876aa39-20ce-49c3-9d52-684d4590bede" containerName="main"
Sep 16 18:49:11.698657 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698658    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b876aa39-20ce-49c3-9d52-684d4590bede" containerName="main"
Sep 16 18:49:11.698869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698668    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerName="istio-proxy"
Sep 16 18:49:11.698869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698674    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerName="istio-proxy"
Sep 16 18:49:11.698869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698679    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60248414-4995-4f51-b233-5dd9f38968a8" containerName="main"
Sep 16 18:49:11.698869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698684    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="60248414-4995-4f51-b233-5dd9f38968a8" containerName="main"
Sep 16 18:49:11.698869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698691    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ef68f3a-a4c3-4b67-9d44-2de265669783" containerName="main"
Sep 16 18:49:11.698869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698696    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ef68f3a-a4c3-4b67-9d44-2de265669783" containerName="main"
Sep 16 18:49:11.698869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698760    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerName="istio-proxy"
Sep 16 18:49:11.698869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698768    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="51121220-40b0-4e10-8976-5e9cd3b01de0" containerName="revoke-keys"
Sep 16 18:49:11.698869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698774    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ef68f3a-a4c3-4b67-9d44-2de265669783" containerName="main"
Sep 16 18:49:11.698869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698780    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee7d27e9-1bbf-429f-b3b2-052f8e704fb3" containerName="main"
Sep 16 18:49:11.698869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698786    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6d40bd-176d-4785-833d-d894891fbe1f" containerName="istio-proxy"
Sep 16 18:49:11.698869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698791    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="60248414-4995-4f51-b233-5dd9f38968a8" containerName="main"
Sep 16 18:49:11.698869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.698796    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b876aa39-20ce-49c3-9d52-684d4590bede" containerName="main"
Sep 16 18:49:11.702017 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.701997    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5"
Sep 16 18:49:11.705094 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.705071    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:49:11.705260 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.705225    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:49:11.722168 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.722124    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5"]
Sep 16 18:49:11.805040 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.805000    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc38b81-d9c3-4e93-8442-846919928c98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5\" (UID: \"ebc38b81-d9c3-4e93-8442-846919928c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5"
Sep 16 18:49:11.905887 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.905854    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc38b81-d9c3-4e93-8442-846919928c98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5\" (UID: \"ebc38b81-d9c3-4e93-8442-846919928c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5"
Sep 16 18:49:11.906481 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:11.906458    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc38b81-d9c3-4e93-8442-846919928c98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5\" (UID: \"ebc38b81-d9c3-4e93-8442-846919928c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5"
Sep 16 18:49:12.013097 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:12.013062    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5"
Sep 16 18:49:12.021536 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:12.021498    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:12.291747 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:12.291717    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5"]
Sep 16 18:49:12.293058 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:49:12.293031    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc38b81_d9c3_4e93_8442_846919928c98.slice/crio-4d1e0a83d82aedd917a72db5fba162dbc785420ac21fa1371b62fc504da90783 WatchSource:0}: Error finding container 4d1e0a83d82aedd917a72db5fba162dbc785420ac21fa1371b62fc504da90783: Status 404 returned error can't find the container with id 4d1e0a83d82aedd917a72db5fba162dbc785420ac21fa1371b62fc504da90783
Sep 16 18:49:12.455216 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:12.455186    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5" event={"ID":"ebc38b81-d9c3-4e93-8442-846919928c98","Type":"ContainerStarted","Data":"4d1e0a83d82aedd917a72db5fba162dbc785420ac21fa1371b62fc504da90783"}
Sep 16 18:49:12.941439 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:12.941341    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:13.464006 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:13.463972    2562 generic.go:356] "Generic (PLEG): container finished" podID="ebc38b81-d9c3-4e93-8442-846919928c98" containerID="0164efb579aebd1a9ac277d288dbf3cfd1021e9814e86a686d533552049ffb96" exitCode=0
Sep 16 18:49:13.464400 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:13.464063    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5" event={"ID":"ebc38b81-d9c3-4e93-8442-846919928c98","Type":"ContainerDied","Data":"0164efb579aebd1a9ac277d288dbf3cfd1021e9814e86a686d533552049ffb96"}
Sep 16 18:49:13.679675 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:13.679646    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:14.529959 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:14.529925    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:14.607507 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:14.607482    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5"
Sep 16 18:49:14.734489 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:14.734394    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc38b81-d9c3-4e93-8442-846919928c98-maas-api-service-ca\") pod \"ebc38b81-d9c3-4e93-8442-846919928c98\" (UID: \"ebc38b81-d9c3-4e93-8442-846919928c98\") "
Sep 16 18:49:14.734841 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:14.734810    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc38b81-d9c3-4e93-8442-846919928c98-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebc38b81-d9c3-4e93-8442-846919928c98" (UID: "ebc38b81-d9c3-4e93-8442-846919928c98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:14.836097 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:14.836062    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc38b81-d9c3-4e93-8442-846919928c98-maas-api-service-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:15.431118 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:15.431078    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:15.477749 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:15.477718    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5"
Sep 16 18:49:15.477950 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:15.477754    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5" event={"ID":"ebc38b81-d9c3-4e93-8442-846919928c98","Type":"ContainerDied","Data":"4d1e0a83d82aedd917a72db5fba162dbc785420ac21fa1371b62fc504da90783"}
Sep 16 18:49:15.477950 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:15.477800    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d1e0a83d82aedd917a72db5fba162dbc785420ac21fa1371b62fc504da90783"
Sep 16 18:49:16.818715 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:16.818684    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:17.354443 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:17.354410    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:17.512929 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:17.512896    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:18.028020 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:18.027985    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:18.438130 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:18.438047    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:18.645406 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:18.645372    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:18.747467 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:18.747432    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:24.255531 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:24.255485    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:27.353210 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:27.353170    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:27.449438 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:27.449399    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:27.537465 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:27.537431    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5"]
Sep 16 18:49:27.539962 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:27.539940    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4dfacb-82lz5"]
Sep 16 18:49:27.541716 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:27.541695    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:27.749194 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:27.749154    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:27.954876 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:27.954830    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:28.051869 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:28.051780    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:28.550529 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:28.550495    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebc38b81-d9c3-4e93-8442-846919928c98" path="/var/lib/kubelet/pods/ebc38b81-d9c3-4e93-8442-846919928c98/volumes"
Sep 16 18:49:28.944986 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:28.944909    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:29.968079 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:29.968042    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:30.715250 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:30.715216    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:49:33.963471 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:33.963436    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:34.067173 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.067133    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:34.200153 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.200118    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"]
Sep 16 18:49:34.200388 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.200363    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f" podUID="f64bc0a9-db3a-4e00-90db-4d81098629b2" containerName="istio-proxy" containerID="cri-o://e623c0b2d09800e1746b996a38efb5377c4210c74b1324e063a4e237b86ef230" gracePeriod=30
Sep 16 18:49:34.234516 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.234445    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:34.594378 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.594291    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b"]
Sep 16 18:49:34.594781 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.594757    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc38b81-d9c3-4e93-8442-846919928c98" containerName="revoke-keys"
Sep 16 18:49:34.594908 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.594785    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc38b81-d9c3-4e93-8442-846919928c98" containerName="revoke-keys"
Sep 16 18:49:34.594908 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.594901    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc38b81-d9c3-4e93-8442-846919928c98" containerName="revoke-keys"
Sep 16 18:49:34.599356 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.599337    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b"
Sep 16 18:49:34.603942 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.603917    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:49:34.617624 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.617598    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b"]
Sep 16 18:49:34.621671 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.621651    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:49:34.715607 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.715574    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d218042-8129-4139-830e-1cf269893f75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b\" (UID: \"8d218042-8129-4139-830e-1cf269893f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b"
Sep 16 18:49:34.816171 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.816130    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d218042-8129-4139-830e-1cf269893f75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b\" (UID: \"8d218042-8129-4139-830e-1cf269893f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b"
Sep 16 18:49:34.816784 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.816765    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d218042-8129-4139-830e-1cf269893f75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b\" (UID: \"8d218042-8129-4139-830e-1cf269893f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b"
Sep 16 18:49:34.836827 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.836794    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:34.909844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:34.909759    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b"
Sep 16 18:49:35.197319 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:35.197292    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b"]
Sep 16 18:49:35.200286 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:49:35.200257    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d218042_8129_4139_830e_1cf269893f75.slice/crio-4c28cf4bf9d81f3724acc081bdea51a1b93badcec7a1c4f340e40884fde22699 WatchSource:0}: Error finding container 4c28cf4bf9d81f3724acc081bdea51a1b93badcec7a1c4f340e40884fde22699: Status 404 returned error can't find the container with id 4c28cf4bf9d81f3724acc081bdea51a1b93badcec7a1c4f340e40884fde22699
Sep 16 18:49:35.604495 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:35.604444    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b" event={"ID":"8d218042-8129-4139-830e-1cf269893f75","Type":"ContainerStarted","Data":"85f58196e3590e58d1e9d044701b0e12a460c795efe6ae48d71b347c88ecdd45"}
Sep 16 18:49:35.604692 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:35.604517    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b" event={"ID":"8d218042-8129-4139-830e-1cf269893f75","Type":"ContainerStarted","Data":"4c28cf4bf9d81f3724acc081bdea51a1b93badcec7a1c4f340e40884fde22699"}
Sep 16 18:49:35.623154 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:35.623107    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b" podStartSLOduration=1.623092646 podStartE2EDuration="1.623092646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:35.62104339 +0000 UTC m=+2013.485947437" watchObservedRunningTime="2026-09-16 18:49:35.623092646 +0000 UTC m=+2013.487996694"
Sep 16 18:49:35.748238 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:35.748159    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:36.477642 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:36.477610    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:36.612651 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:36.612611    2562 generic.go:356] "Generic (PLEG): container finished" podID="8d218042-8129-4139-830e-1cf269893f75" containerID="85f58196e3590e58d1e9d044701b0e12a460c795efe6ae48d71b347c88ecdd45" exitCode=0
Sep 16 18:49:36.612860 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:36.612703    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b" event={"ID":"8d218042-8129-4139-830e-1cf269893f75","Type":"ContainerDied","Data":"85f58196e3590e58d1e9d044701b0e12a460c795efe6ae48d71b347c88ecdd45"}
Sep 16 18:49:37.312467 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:37.312436    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:37.753553 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:37.753515    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b"
Sep 16 18:49:37.944602 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:37.944517    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d218042-8129-4139-830e-1cf269893f75-maas-api-service-ca\") pod \"8d218042-8129-4139-830e-1cf269893f75\" (UID: \"8d218042-8129-4139-830e-1cf269893f75\") "
Sep 16 18:49:37.944841 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:37.944819    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d218042-8129-4139-830e-1cf269893f75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d218042-8129-4139-830e-1cf269893f75" (UID: "8d218042-8129-4139-830e-1cf269893f75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:38.045877 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:38.045846    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d218042-8129-4139-830e-1cf269893f75-maas-api-service-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:38.628118 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:38.628090    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b"
Sep 16 18:49:38.628256 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:38.628088    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b" event={"ID":"8d218042-8129-4139-830e-1cf269893f75","Type":"ContainerDied","Data":"4c28cf4bf9d81f3724acc081bdea51a1b93badcec7a1c4f340e40884fde22699"}
Sep 16 18:49:38.628256 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:38.628221    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c28cf4bf9d81f3724acc081bdea51a1b93badcec7a1c4f340e40884fde22699"
Sep 16 18:49:39.319331 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.319306    2562 patch_prober.go:28] interesting pod/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 16 18:49:39.319650 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.319369    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f" podUID="f64bc0a9-db3a-4e00-90db-4d81098629b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 16 18:49:39.461381 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.461352    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:49:39.642229 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.642143    2562 generic.go:356] "Generic (PLEG): container finished" podID="f64bc0a9-db3a-4e00-90db-4d81098629b2" containerID="e623c0b2d09800e1746b996a38efb5377c4210c74b1324e063a4e237b86ef230" exitCode=0
Sep 16 18:49:39.642229 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.642210    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"
Sep 16 18:49:39.642408 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.642224    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f" event={"ID":"f64bc0a9-db3a-4e00-90db-4d81098629b2","Type":"ContainerDied","Data":"e623c0b2d09800e1746b996a38efb5377c4210c74b1324e063a4e237b86ef230"}
Sep 16 18:49:39.642408 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.642303    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f" event={"ID":"f64bc0a9-db3a-4e00-90db-4d81098629b2","Type":"ContainerDied","Data":"c5e8f0ee9777e24752f243b04319082d9b25068c9ba28f43fc2410de60e63333"}
Sep 16 18:49:39.642408 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.642325    2562 scope.go:117] "RemoveContainer" containerID="e623c0b2d09800e1746b996a38efb5377c4210c74b1324e063a4e237b86ef230"
Sep 16 18:49:39.652011 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.651989    2562 scope.go:117] "RemoveContainer" containerID="c39bb2b6f3dc815760e5bc7962253cfc75dee5d48533d4891ad86d3b9c597c8a"
Sep 16 18:49:39.660207 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.660187    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-envoy\") pod \"f64bc0a9-db3a-4e00-90db-4d81098629b2\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") "
Sep 16 18:49:39.660323 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.660235    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-data\") pod \"f64bc0a9-db3a-4e00-90db-4d81098629b2\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") "
Sep 16 18:49:39.660323 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.660287    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxxnv\" (UniqueName: \"kubernetes.io/projected/f64bc0a9-db3a-4e00-90db-4d81098629b2-kube-api-access-sxxnv\") pod \"f64bc0a9-db3a-4e00-90db-4d81098629b2\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") "
Sep 16 18:49:39.660509 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.660331    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-podinfo\") pod \"f64bc0a9-db3a-4e00-90db-4d81098629b2\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") "
Sep 16 18:49:39.660509 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.660363    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-credential-socket\") pod \"f64bc0a9-db3a-4e00-90db-4d81098629b2\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") "
Sep 16 18:49:39.660509 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.660383    2562 scope.go:117] "RemoveContainer" containerID="e623c0b2d09800e1746b996a38efb5377c4210c74b1324e063a4e237b86ef230"
Sep 16 18:49:39.660509 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.660388    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-token\") pod \"f64bc0a9-db3a-4e00-90db-4d81098629b2\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") "
Sep 16 18:49:39.660754 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.660569    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-data" (OuterVolumeSpecName: "istio-data") pod "f64bc0a9-db3a-4e00-90db-4d81098629b2" (UID: "f64bc0a9-db3a-4e00-90db-4d81098629b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:39.660754 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:49:39.660721    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e623c0b2d09800e1746b996a38efb5377c4210c74b1324e063a4e237b86ef230\": container with ID starting with e623c0b2d09800e1746b996a38efb5377c4210c74b1324e063a4e237b86ef230 not found: ID does not exist" containerID="e623c0b2d09800e1746b996a38efb5377c4210c74b1324e063a4e237b86ef230"
Sep 16 18:49:39.660862 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.660757    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e623c0b2d09800e1746b996a38efb5377c4210c74b1324e063a4e237b86ef230"} err="failed to get container status \"e623c0b2d09800e1746b996a38efb5377c4210c74b1324e063a4e237b86ef230\": rpc error: code = NotFound desc = could not find container \"e623c0b2d09800e1746b996a38efb5377c4210c74b1324e063a4e237b86ef230\": container with ID starting with e623c0b2d09800e1746b996a38efb5377c4210c74b1324e063a4e237b86ef230 not found: ID does not exist"
Sep 16 18:49:39.660862 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.660782    2562 scope.go:117] "RemoveContainer" containerID="c39bb2b6f3dc815760e5bc7962253cfc75dee5d48533d4891ad86d3b9c597c8a"
Sep 16 18:49:39.660976 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.660927    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f64bc0a9-db3a-4e00-90db-4d81098629b2" (UID: "f64bc0a9-db3a-4e00-90db-4d81098629b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:39.661039 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.661021    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-workload-certs\") pod \"f64bc0a9-db3a-4e00-90db-4d81098629b2\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") "
Sep 16 18:49:39.661095 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:49:39.661068    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39bb2b6f3dc815760e5bc7962253cfc75dee5d48533d4891ad86d3b9c597c8a\": container with ID starting with c39bb2b6f3dc815760e5bc7962253cfc75dee5d48533d4891ad86d3b9c597c8a not found: ID does not exist" containerID="c39bb2b6f3dc815760e5bc7962253cfc75dee5d48533d4891ad86d3b9c597c8a"
Sep 16 18:49:39.661095 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.661080    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f64bc0a9-db3a-4e00-90db-4d81098629b2-istiod-ca-cert\") pod \"f64bc0a9-db3a-4e00-90db-4d81098629b2\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") "
Sep 16 18:49:39.661198 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.661097    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39bb2b6f3dc815760e5bc7962253cfc75dee5d48533d4891ad86d3b9c597c8a"} err="failed to get container status \"c39bb2b6f3dc815760e5bc7962253cfc75dee5d48533d4891ad86d3b9c597c8a\": rpc error: code = NotFound desc = could not find container \"c39bb2b6f3dc815760e5bc7962253cfc75dee5d48533d4891ad86d3b9c597c8a\": container with ID starting with c39bb2b6f3dc815760e5bc7962253cfc75dee5d48533d4891ad86d3b9c597c8a not found: ID does not exist"
Sep 16 18:49:39.661198 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.661142    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-workload-socket\") pod \"f64bc0a9-db3a-4e00-90db-4d81098629b2\" (UID: \"f64bc0a9-db3a-4e00-90db-4d81098629b2\") "
Sep 16 18:49:39.661471 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.661452    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-data\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:39.661627 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.661611    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-credential-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:39.661693 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.661572    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f64bc0a9-db3a-4e00-90db-4d81098629b2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f64bc0a9-db3a-4e00-90db-4d81098629b2" (UID: "f64bc0a9-db3a-4e00-90db-4d81098629b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:39.661693 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.661594    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f64bc0a9-db3a-4e00-90db-4d81098629b2" (UID: "f64bc0a9-db3a-4e00-90db-4d81098629b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:39.661693 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.661657    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f64bc0a9-db3a-4e00-90db-4d81098629b2" (UID: "f64bc0a9-db3a-4e00-90db-4d81098629b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:39.662836 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.662809    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f64bc0a9-db3a-4e00-90db-4d81098629b2" (UID: "f64bc0a9-db3a-4e00-90db-4d81098629b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:39.662937 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.662917    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64bc0a9-db3a-4e00-90db-4d81098629b2-kube-api-access-sxxnv" (OuterVolumeSpecName: "kube-api-access-sxxnv") pod "f64bc0a9-db3a-4e00-90db-4d81098629b2" (UID: "f64bc0a9-db3a-4e00-90db-4d81098629b2"). InnerVolumeSpecName "kube-api-access-sxxnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:39.663073 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.663049    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f64bc0a9-db3a-4e00-90db-4d81098629b2" (UID: "f64bc0a9-db3a-4e00-90db-4d81098629b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:49:39.663126 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.663096    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-token" (OuterVolumeSpecName: "istio-token") pod "f64bc0a9-db3a-4e00-90db-4d81098629b2" (UID: "f64bc0a9-db3a-4e00-90db-4d81098629b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:39.762337 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.762310    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f64bc0a9-db3a-4e00-90db-4d81098629b2-istiod-ca-cert\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:39.762337 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.762334    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-workload-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:39.762337 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.762342    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-envoy\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:39.762571 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.762351    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxxnv\" (UniqueName: \"kubernetes.io/projected/f64bc0a9-db3a-4e00-90db-4d81098629b2-kube-api-access-sxxnv\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:39.762571 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.762359    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-podinfo\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:39.762571 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.762367    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f64bc0a9-db3a-4e00-90db-4d81098629b2-istio-token\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:39.762571 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.762375    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f64bc0a9-db3a-4e00-90db-4d81098629b2-workload-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:39.969664 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.969631    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"]
Sep 16 18:49:39.977664 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:39.977634    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4dfacb-openshift-default-68989dccf4-lqb7f"]
Sep 16 18:49:40.550884 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:40.550849    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f64bc0a9-db3a-4e00-90db-4d81098629b2" path="/var/lib/kubelet/pods/f64bc0a9-db3a-4e00-90db-4d81098629b2/volumes"
Sep 16 18:49:45.325471 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:45.325440    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:45.442633 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:45.442595    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:45.542975 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:45.542938    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:45.608318 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:45.608226    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b"]
Sep 16 18:49:45.612033 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:45.612008    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-04899e-8nj9b"]
Sep 16 18:49:45.616500 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:45.616476    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:45.950781 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:45.950736    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:46.031286 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:46.031246    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:46.550695 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:46.550665    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d218042-8129-4139-830e-1cf269893f75" path="/var/lib/kubelet/pods/8d218042-8129-4139-830e-1cf269893f75/volumes"
Sep 16 18:49:46.930781 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:46.930673    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:48.326972 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:48.326936    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:49:52.664601 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:52.664557    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:52.735648 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:52.735606    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:52.963745 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:52.963707    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:53.289974 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.289893    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg"]
Sep 16 18:49:53.290315 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.290300    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d218042-8129-4139-830e-1cf269893f75" containerName="revoke-keys"
Sep 16 18:49:53.290361 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.290317    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d218042-8129-4139-830e-1cf269893f75" containerName="revoke-keys"
Sep 16 18:49:53.290361 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.290328    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f64bc0a9-db3a-4e00-90db-4d81098629b2" containerName="istio-proxy"
Sep 16 18:49:53.290361 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.290333    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f64bc0a9-db3a-4e00-90db-4d81098629b2" containerName="istio-proxy"
Sep 16 18:49:53.290361 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.290340    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f64bc0a9-db3a-4e00-90db-4d81098629b2" containerName="istio-proxy"
Sep 16 18:49:53.290361 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.290347    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f64bc0a9-db3a-4e00-90db-4d81098629b2" containerName="istio-proxy"
Sep 16 18:49:53.290515 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.290402    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d218042-8129-4139-830e-1cf269893f75" containerName="revoke-keys"
Sep 16 18:49:53.290515 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.290411    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f64bc0a9-db3a-4e00-90db-4d81098629b2" containerName="istio-proxy"
Sep 16 18:49:53.290515 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.290419    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f64bc0a9-db3a-4e00-90db-4d81098629b2" containerName="istio-proxy"
Sep 16 18:49:53.294975 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.294959    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg"
Sep 16 18:49:53.297560 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.297519    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgwrl\""
Sep 16 18:49:53.297689 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.297571    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 18:49:53.303736 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.303710    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg"]
Sep 16 18:49:53.374975 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.374942    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71be3ea0-2deb-4d47-881e-d92e54809d97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg\" (UID: \"71be3ea0-2deb-4d47-881e-d92e54809d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg"
Sep 16 18:49:53.459021 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.458978    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:53.475866 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.475835    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71be3ea0-2deb-4d47-881e-d92e54809d97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg\" (UID: \"71be3ea0-2deb-4d47-881e-d92e54809d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg"
Sep 16 18:49:53.476413 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.476393    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71be3ea0-2deb-4d47-881e-d92e54809d97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg\" (UID: \"71be3ea0-2deb-4d47-881e-d92e54809d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg"
Sep 16 18:49:53.606123 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.606038    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg"
Sep 16 18:49:53.878955 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:53.878927    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg"]
Sep 16 18:49:53.880142 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:49:53.880113    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71be3ea0_2deb_4d47_881e_d92e54809d97.slice/crio-8705b75142cd61b91aa58bfd1eaf86b79dc0d208de66e99277561500c835aaa0 WatchSource:0}: Error finding container 8705b75142cd61b91aa58bfd1eaf86b79dc0d208de66e99277561500c835aaa0: Status 404 returned error can't find the container with id 8705b75142cd61b91aa58bfd1eaf86b79dc0d208de66e99277561500c835aaa0
Sep 16 18:49:54.258596 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:54.258559    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:54.740281 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:54.740245    2562 generic.go:356] "Generic (PLEG): container finished" podID="71be3ea0-2deb-4d47-881e-d92e54809d97" containerID="7f1cf7c6ffa61a9937034cde9516b71e164935cb846b0a6ab396269dcab18b50" exitCode=0
Sep 16 18:49:54.740446 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:54.740332    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg" event={"ID":"71be3ea0-2deb-4d47-881e-d92e54809d97","Type":"ContainerDied","Data":"7f1cf7c6ffa61a9937034cde9516b71e164935cb846b0a6ab396269dcab18b50"}
Sep 16 18:49:54.740446 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:54.740385    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg" event={"ID":"71be3ea0-2deb-4d47-881e-d92e54809d97","Type":"ContainerStarted","Data":"8705b75142cd61b91aa58bfd1eaf86b79dc0d208de66e99277561500c835aaa0"}
Sep 16 18:49:55.057078 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:55.057045    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:55.877734 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:55.877710    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg"
Sep 16 18:49:56.001424 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:56.001350    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71be3ea0-2deb-4d47-881e-d92e54809d97-maas-api-service-ca\") pod \"71be3ea0-2deb-4d47-881e-d92e54809d97\" (UID: \"71be3ea0-2deb-4d47-881e-d92e54809d97\") "
Sep 16 18:49:56.001710 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:56.001687    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71be3ea0-2deb-4d47-881e-d92e54809d97-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71be3ea0-2deb-4d47-881e-d92e54809d97" (UID: "71be3ea0-2deb-4d47-881e-d92e54809d97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:56.102878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:56.102844    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71be3ea0-2deb-4d47-881e-d92e54809d97-maas-api-service-ca\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:49:56.760829 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:56.760799    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg"
Sep 16 18:49:56.761000 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:56.760795    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg" event={"ID":"71be3ea0-2deb-4d47-881e-d92e54809d97","Type":"ContainerDied","Data":"8705b75142cd61b91aa58bfd1eaf86b79dc0d208de66e99277561500c835aaa0"}
Sep 16 18:49:56.761000 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:49:56.760941    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8705b75142cd61b91aa58bfd1eaf86b79dc0d208de66e99277561500c835aaa0"
Sep 16 18:50:03.815375 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:03.815338    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:03.817444 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:03.817416    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-520999-29826405-59lzx"]
Sep 16 18:50:03.820017 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:03.819994    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-520999-29826405-59lzx"]
Sep 16 18:50:03.910138 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:03.910094    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:04.010821 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:04.010790    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:04.042965 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:04.042935    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg"]
Sep 16 18:50:04.046649 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:04.046627    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-520999-kxxcg"]
Sep 16 18:50:04.086735 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:04.086664    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:04.404268 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:04.404164    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:04.550891 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:04.550857    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d858b9e-8c3f-4d62-b772-a0e525e34117" path="/var/lib/kubelet/pods/3d858b9e-8c3f-4d62-b772-a0e525e34117/volumes"
Sep 16 18:50:04.551206 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:04.551191    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71be3ea0-2deb-4d47-881e-d92e54809d97" path="/var/lib/kubelet/pods/71be3ea0-2deb-4d47-881e-d92e54809d97/volumes"
Sep 16 18:50:04.588622 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:04.588595    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:06.323615 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:06.323582    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:50:10.861355 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:10.861326    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:10.965484 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:10.965448    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:11.147988 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:11.147909    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"]
Sep 16 18:50:11.148205 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:11.148172    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx" podUID="fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" containerName="istio-proxy" containerID="cri-o://32ac34a6faa390a81ac1442063270fd9b4b9ca46105973ed9e17f771257493a6" gracePeriod=30
Sep 16 18:50:11.190094 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:11.190051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:16.395804 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.395780    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:50:16.490730 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.490700    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istiod-ca-cert\") pod \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") "
Sep 16 18:50:16.490896 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.490739    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-credential-socket\") pod \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") "
Sep 16 18:50:16.490896 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.490774    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-workload-socket\") pod \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") "
Sep 16 18:50:16.490896 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.490794    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-token\") pod \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") "
Sep 16 18:50:16.490896 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.490831    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-podinfo\") pod \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") "
Sep 16 18:50:16.490896 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.490848    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6ttr\" (UniqueName: \"kubernetes.io/projected/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-kube-api-access-v6ttr\") pod \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") "
Sep 16 18:50:16.490896 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.490876    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-data\") pod \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") "
Sep 16 18:50:16.491251 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.490940    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-envoy\") pod \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") "
Sep 16 18:50:16.491251 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.490974    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-workload-certs\") pod \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\" (UID: \"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6\") "
Sep 16 18:50:16.491251 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.491064    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" (UID: "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:16.491251 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.491130    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" (UID: "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:16.491251 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.491202    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" (UID: "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:16.491519 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.491307    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istiod-ca-cert\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:50:16.491519 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.491326    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-credential-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:50:16.491519 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.491341    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-workload-socket\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:50:16.491519 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.491379    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-data" (OuterVolumeSpecName: "istio-data") pod "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" (UID: "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:16.491729 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.491538    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" (UID: "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:16.493168 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.493142    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" (UID: "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:50:16.493346 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.493231    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" (UID: "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:16.493571 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.493527    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-kube-api-access-v6ttr" (OuterVolumeSpecName: "kube-api-access-v6ttr") pod "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" (UID: "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6"). InnerVolumeSpecName "kube-api-access-v6ttr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:16.493707 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.493566    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" (UID: "fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:16.592769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.592744    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-envoy\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:50:16.592769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.592770    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-workload-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:50:16.592941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.592778    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-token\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:50:16.592941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.592787    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-podinfo\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:50:16.592941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.592796    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6ttr\" (UniqueName: \"kubernetes.io/projected/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-kube-api-access-v6ttr\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:50:16.592941 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.592804    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6-istio-data\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:50:16.892964 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.892874    2562 generic.go:356] "Generic (PLEG): container finished" podID="fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" containerID="32ac34a6faa390a81ac1442063270fd9b4b9ca46105973ed9e17f771257493a6" exitCode=0
Sep 16 18:50:16.892964 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.892922    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"
Sep 16 18:50:16.893156 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.892994    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx" event={"ID":"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6","Type":"ContainerDied","Data":"32ac34a6faa390a81ac1442063270fd9b4b9ca46105973ed9e17f771257493a6"}
Sep 16 18:50:16.893156 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.893046    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx" event={"ID":"fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6","Type":"ContainerDied","Data":"0f6c4b93a392a5b8cbd04986734cfca2b1b403702d864fdb56422a5dd64adccb"}
Sep 16 18:50:16.893156 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.893068    2562 scope.go:117] "RemoveContainer" containerID="32ac34a6faa390a81ac1442063270fd9b4b9ca46105973ed9e17f771257493a6"
Sep 16 18:50:16.904803 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.904782    2562 scope.go:117] "RemoveContainer" containerID="56e059e23d05a7f981ced60a5e017f8554d92416ade4e684ad0671970627e1a2"
Sep 16 18:50:16.913632 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.913614    2562 scope.go:117] "RemoveContainer" containerID="32ac34a6faa390a81ac1442063270fd9b4b9ca46105973ed9e17f771257493a6"
Sep 16 18:50:16.913964 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:50:16.913932    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ac34a6faa390a81ac1442063270fd9b4b9ca46105973ed9e17f771257493a6\": container with ID starting with 32ac34a6faa390a81ac1442063270fd9b4b9ca46105973ed9e17f771257493a6 not found: ID does not exist" containerID="32ac34a6faa390a81ac1442063270fd9b4b9ca46105973ed9e17f771257493a6"
Sep 16 18:50:16.914064 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.913973    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ac34a6faa390a81ac1442063270fd9b4b9ca46105973ed9e17f771257493a6"} err="failed to get container status \"32ac34a6faa390a81ac1442063270fd9b4b9ca46105973ed9e17f771257493a6\": rpc error: code = NotFound desc = could not find container \"32ac34a6faa390a81ac1442063270fd9b4b9ca46105973ed9e17f771257493a6\": container with ID starting with 32ac34a6faa390a81ac1442063270fd9b4b9ca46105973ed9e17f771257493a6 not found: ID does not exist"
Sep 16 18:50:16.914064 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.913996    2562 scope.go:117] "RemoveContainer" containerID="56e059e23d05a7f981ced60a5e017f8554d92416ade4e684ad0671970627e1a2"
Sep 16 18:50:16.914273 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:50:16.914252    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e059e23d05a7f981ced60a5e017f8554d92416ade4e684ad0671970627e1a2\": container with ID starting with 56e059e23d05a7f981ced60a5e017f8554d92416ade4e684ad0671970627e1a2 not found: ID does not exist" containerID="56e059e23d05a7f981ced60a5e017f8554d92416ade4e684ad0671970627e1a2"
Sep 16 18:50:16.914330 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.914283    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e059e23d05a7f981ced60a5e017f8554d92416ade4e684ad0671970627e1a2"} err="failed to get container status \"56e059e23d05a7f981ced60a5e017f8554d92416ade4e684ad0671970627e1a2\": rpc error: code = NotFound desc = could not find container \"56e059e23d05a7f981ced60a5e017f8554d92416ade4e684ad0671970627e1a2\": container with ID starting with 56e059e23d05a7f981ced60a5e017f8554d92416ade4e684ad0671970627e1a2 not found: ID does not exist"
Sep 16 18:50:16.915174 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.915155    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"]
Sep 16 18:50:16.921684 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:16.921664    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-520999-openshift-default-5559fcc6d8-zbzlx"]
Sep 16 18:50:17.135450 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:17.135409    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:50:18.550769 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:50:18.550736    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" path="/var/lib/kubelet/pods/fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6/volumes"
Sep 16 18:51:02.744818 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:51:02.744791    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:51:02.749244 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:51:02.749216    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:51:02.751627 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:51:02.751605    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:51:02.755424 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:51:02.755406    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:51:03.099644 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:51:03.099563    2562 scope.go:117] "RemoveContainer" containerID="a1096d1b9b4bae09d70ebb7d0a4f53cf9e124862bc1b432978155d22d62a0461"
Sep 16 18:51:03.108618 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:51:03.108599    2562 scope.go:117] "RemoveContainer" containerID="bcff8b6db53a8ed15e41397a1a7858f6bdf9d8b2c4893552884916e1118cdd1d"
Sep 16 18:51:03.117191 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:51:03.117172    2562 scope.go:117] "RemoveContainer" containerID="9118e0705004d7d4878eb4964f5e33b5773202b5aef95830907bca6f47598fcd"
Sep 16 18:51:03.125144 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:51:03.125124    2562 scope.go:117] "RemoveContainer" containerID="8d6a926f569a7d58a87bc2f4a86f070052f9ce4909d867fa05657c8b37fb52dd"
Sep 16 18:51:03.132814 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:51:03.132793    2562 scope.go:117] "RemoveContainer" containerID="9767cf859b1fc0adef47c8485e41260af217d61596713f30812969cad3978fa9"
Sep 16 18:51:40.146498 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:51:40.146458    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:51:41.044702 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:51:41.044666    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:42.494183 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:51:42.494146    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:03.176604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:03.176572    2562 scope.go:117] "RemoveContainer" containerID="8ea9b950df8b7e6b48426a9f8316bfba50d528764499ea778a87ed1243de73ff"
Sep 16 18:52:03.185128 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:03.185097    2562 scope.go:117] "RemoveContainer" containerID="91da309506063b32dfd6f75f307c9787892cb6f0d91b6e5d6357760404207404"
Sep 16 18:52:03.196153 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:03.196125    2562 scope.go:117] "RemoveContainer" containerID="cd306b387da7cc7a64ecc1b43335538d21ada56a10044421582f75cbf3bf6a18"
Sep 16 18:52:05.820406 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:05.820368    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:52:13.511080 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:13.511038    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:52:21.416611 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:21.416573    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:52:39.819979 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:39.819946    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:52:43.789195 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:43.789139    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:45.752196 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:45.752163    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:52:47.308393 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:47.308354    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"]
Sep 16 18:52:47.308879 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:47.308566    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx" podUID="d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce" containerName="manager" containerID="cri-o://0fd3499234483d013aadf94b87798104bedc354aa185aec447779a37826184ab" gracePeriod=10
Sep 16 18:52:49.354331 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.354304    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"
Sep 16 18:52:49.481903 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.481869    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce-extensions-socket-volume\") pod \"d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce\" (UID: \"d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce\") "
Sep 16 18:52:49.482120 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.481935    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gns7l\" (UniqueName: \"kubernetes.io/projected/d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce-kube-api-access-gns7l\") pod \"d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce\" (UID: \"d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce\") "
Sep 16 18:52:49.482318 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.482295    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce" (UID: "d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:49.484018 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.483994    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce-kube-api-access-gns7l" (OuterVolumeSpecName: "kube-api-access-gns7l") pod "d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce" (UID: "d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce"). InnerVolumeSpecName "kube-api-access-gns7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:49.582606 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.582571    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce-extensions-socket-volume\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:52:49.582606 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.582598    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gns7l\" (UniqueName: \"kubernetes.io/projected/d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce-kube-api-access-gns7l\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:52:49.813804 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.813716    2562 generic.go:356] "Generic (PLEG): container finished" podID="d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce" containerID="0fd3499234483d013aadf94b87798104bedc354aa185aec447779a37826184ab" exitCode=0
Sep 16 18:52:49.813804 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.813756    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"
Sep 16 18:52:49.813998 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.813800    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx" event={"ID":"d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce","Type":"ContainerDied","Data":"0fd3499234483d013aadf94b87798104bedc354aa185aec447779a37826184ab"}
Sep 16 18:52:49.813998 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.813848    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx" event={"ID":"d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce","Type":"ContainerDied","Data":"c9308a35aa5eb660aa50b27f74d899ac619a9936a70fe4f37277159642ec9b2c"}
Sep 16 18:52:49.813998 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.813870    2562 scope.go:117] "RemoveContainer" containerID="0fd3499234483d013aadf94b87798104bedc354aa185aec447779a37826184ab"
Sep 16 18:52:49.823439 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.823421    2562 scope.go:117] "RemoveContainer" containerID="0fd3499234483d013aadf94b87798104bedc354aa185aec447779a37826184ab"
Sep 16 18:52:49.823738 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:52:49.823721    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd3499234483d013aadf94b87798104bedc354aa185aec447779a37826184ab\": container with ID starting with 0fd3499234483d013aadf94b87798104bedc354aa185aec447779a37826184ab not found: ID does not exist" containerID="0fd3499234483d013aadf94b87798104bedc354aa185aec447779a37826184ab"
Sep 16 18:52:49.823797 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.823748    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd3499234483d013aadf94b87798104bedc354aa185aec447779a37826184ab"} err="failed to get container status \"0fd3499234483d013aadf94b87798104bedc354aa185aec447779a37826184ab\": rpc error: code = NotFound desc = could not find container \"0fd3499234483d013aadf94b87798104bedc354aa185aec447779a37826184ab\": container with ID starting with 0fd3499234483d013aadf94b87798104bedc354aa185aec447779a37826184ab not found: ID does not exist"
Sep 16 18:52:49.837070 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.837043    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"]
Sep 16 18:52:49.840472 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:49.840449    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxgnx"]
Sep 16 18:52:50.551494 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:52:50.551461    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce" path="/var/lib/kubelet/pods/d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce/volumes"
Sep 16 18:53:00.358508 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:00.358468    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:00.454519 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:00.454485    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:03.225853 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:03.225820    2562 scope.go:117] "RemoveContainer" containerID="b5daa2eaf058f11ef8ce46f120576cd89a7197201f0051bf3b13480f192e4436"
Sep 16 18:53:17.639465 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:17.639427    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:47.983125 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:47.983084    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w"]
Sep 16 18:53:47.983588 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:47.983440    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" containerName="istio-proxy"
Sep 16 18:53:47.983588 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:47.983452    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" containerName="istio-proxy"
Sep 16 18:53:47.983588 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:47.983462    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" containerName="istio-proxy"
Sep 16 18:53:47.983588 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:47.983467    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" containerName="istio-proxy"
Sep 16 18:53:47.983588 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:47.983479    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce" containerName="manager"
Sep 16 18:53:47.983588 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:47.983484    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce" containerName="manager"
Sep 16 18:53:47.983588 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:47.983490    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71be3ea0-2deb-4d47-881e-d92e54809d97" containerName="revoke-keys"
Sep 16 18:53:47.983588 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:47.983495    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="71be3ea0-2deb-4d47-881e-d92e54809d97" containerName="revoke-keys"
Sep 16 18:53:47.983844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:47.983600    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" containerName="istio-proxy"
Sep 16 18:53:47.983844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:47.983611    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd2d03b1-40e6-4025-9c8e-f63bc2ce9de6" containerName="istio-proxy"
Sep 16 18:53:47.983844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:47.983623    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="71be3ea0-2deb-4d47-881e-d92e54809d97" containerName="revoke-keys"
Sep 16 18:53:47.983844 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:47.983635    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6c53cd6-2a90-4e31-b726-dd79cdc5f4ce" containerName="manager"
Sep 16 18:53:47.986690 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:47.986669    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w"
Sep 16 18:53:47.990163 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:47.990133    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dfm9g\""
Sep 16 18:53:48.002068 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:48.002035    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w"]
Sep 16 18:53:48.110273 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:48.110230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bafd386-d265-4652-b16b-f6a82bbbb02a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-shb2w\" (UID: \"0bafd386-d265-4652-b16b-f6a82bbbb02a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w"
Sep 16 18:53:48.110467 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:48.110311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvpb9\" (UniqueName: \"kubernetes.io/projected/0bafd386-d265-4652-b16b-f6a82bbbb02a-kube-api-access-kvpb9\") pod \"kuadrant-operator-controller-manager-5769f99d9f-shb2w\" (UID: \"0bafd386-d265-4652-b16b-f6a82bbbb02a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w"
Sep 16 18:53:48.211441 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:48.211406    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvpb9\" (UniqueName: \"kubernetes.io/projected/0bafd386-d265-4652-b16b-f6a82bbbb02a-kube-api-access-kvpb9\") pod \"kuadrant-operator-controller-manager-5769f99d9f-shb2w\" (UID: \"0bafd386-d265-4652-b16b-f6a82bbbb02a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w"
Sep 16 18:53:48.211641 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:48.211480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bafd386-d265-4652-b16b-f6a82bbbb02a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-shb2w\" (UID: \"0bafd386-d265-4652-b16b-f6a82bbbb02a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w"
Sep 16 18:53:48.211878 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:48.211860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bafd386-d265-4652-b16b-f6a82bbbb02a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-shb2w\" (UID: \"0bafd386-d265-4652-b16b-f6a82bbbb02a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w"
Sep 16 18:53:48.225810 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:48.225770    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvpb9\" (UniqueName: \"kubernetes.io/projected/0bafd386-d265-4652-b16b-f6a82bbbb02a-kube-api-access-kvpb9\") pod \"kuadrant-operator-controller-manager-5769f99d9f-shb2w\" (UID: \"0bafd386-d265-4652-b16b-f6a82bbbb02a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w"
Sep 16 18:53:48.300596 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:48.300516    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w"
Sep 16 18:53:48.583065 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:48.583032    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w"]
Sep 16 18:53:48.585813 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:53:48.585784    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bafd386_d265_4652_b16b_f6a82bbbb02a.slice/crio-3cbc483a4c735c9c190345427cbf6594268a50bfe4f927eb0e0f05838e8276d6 WatchSource:0}: Error finding container 3cbc483a4c735c9c190345427cbf6594268a50bfe4f927eb0e0f05838e8276d6: Status 404 returned error can't find the container with id 3cbc483a4c735c9c190345427cbf6594268a50bfe4f927eb0e0f05838e8276d6
Sep 16 18:53:49.164796 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:49.164758    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w" event={"ID":"0bafd386-d265-4652-b16b-f6a82bbbb02a","Type":"ContainerStarted","Data":"279391d3124a342f6afc86dcb97e955faa0c0072a5600100a653b960c2c49a9b"}
Sep 16 18:53:49.165218 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:49.164825    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w" event={"ID":"0bafd386-d265-4652-b16b-f6a82bbbb02a","Type":"ContainerStarted","Data":"3cbc483a4c735c9c190345427cbf6594268a50bfe4f927eb0e0f05838e8276d6"}
Sep 16 18:53:49.165218 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:49.164948    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w"
Sep 16 18:53:49.207098 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:53:49.207035    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w" podStartSLOduration=2.207016339 podStartE2EDuration="2.207016339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:49.202920299 +0000 UTC m=+2267.067824347" watchObservedRunningTime="2026-09-16 18:53:49.207016339 +0000 UTC m=+2267.071920387"
Sep 16 18:54:00.173234 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:00.173199    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shb2w"
Sep 16 18:54:12.317909 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:12.317871    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:54:21.206973 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:21.206930    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:54:27.936798 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:27.936764    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"]
Sep 16 18:54:27.940074 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:27.940058    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:27.944027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:27.943996    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0583fa89-kserve-self-signed-certs\""
Sep 16 18:54:27.955933 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:27.955903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-home\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:27.956073 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:27.955950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-model-cache\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:27.956073 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:27.955998    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-dshm\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:27.956073 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:27.956039    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-tmp-dir\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:27.956209 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:27.956103    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdb9f\" (UniqueName: \"kubernetes.io/projected/4f73ff3d-accf-402e-b781-bb82d2dc6a55-kube-api-access-jdb9f\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:27.956209 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:27.956152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f73ff3d-accf-402e-b781-bb82d2dc6a55-tls-certs\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:27.958822 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:27.958793    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"]
Sep 16 18:54:27.968612 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:27.968587    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:28.057230 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:28.057191    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-home\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:28.057230 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:28.057231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-model-cache\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:28.057476 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:28.057255    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-dshm\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:28.057476 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:28.057277    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-tmp-dir\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:28.057476 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:28.057313    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdb9f\" (UniqueName: \"kubernetes.io/projected/4f73ff3d-accf-402e-b781-bb82d2dc6a55-kube-api-access-jdb9f\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:28.057476 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:28.057333    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f73ff3d-accf-402e-b781-bb82d2dc6a55-tls-certs\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:28.057762 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:28.057718    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-home\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:28.057823 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:28.057755    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-model-cache\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:28.057823 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:28.057790    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-tmp-dir\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:28.059723 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:28.059700    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-dshm\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:28.059993 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:28.059975    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f73ff3d-accf-402e-b781-bb82d2dc6a55-tls-certs\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:28.068717 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:28.068691    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdb9f\" (UniqueName: \"kubernetes.io/projected/4f73ff3d-accf-402e-b781-bb82d2dc6a55-kube-api-access-jdb9f\") pod \"e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") " pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:28.248984 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:28.248945    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:28.533863 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:28.533835    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"]
Sep 16 18:54:28.535329 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:54:28.535303    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f73ff3d_accf_402e_b781_bb82d2dc6a55.slice/crio-d8521efa524e2f0a460c9953b55aabe0f64fe027afc6a6b7ef6af1f274174416 WatchSource:0}: Error finding container d8521efa524e2f0a460c9953b55aabe0f64fe027afc6a6b7ef6af1f274174416: Status 404 returned error can't find the container with id d8521efa524e2f0a460c9953b55aabe0f64fe027afc6a6b7ef6af1f274174416
Sep 16 18:54:29.406096 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:29.406061    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt" event={"ID":"4f73ff3d-accf-402e-b781-bb82d2dc6a55","Type":"ContainerStarted","Data":"11fa4c54eb9aee449201d4c06107f8cfc77f0f4498a4a46de426bd07442baaa2"}
Sep 16 18:54:29.406096 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:29.406102    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt" event={"ID":"4f73ff3d-accf-402e-b781-bb82d2dc6a55","Type":"ContainerStarted","Data":"d8521efa524e2f0a460c9953b55aabe0f64fe027afc6a6b7ef6af1f274174416"}
Sep 16 18:54:29.427356 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:29.427301    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt" podStartSLOduration=2.427281662 podStartE2EDuration="2.427281662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:29.426147714 +0000 UTC m=+2307.291051763" watchObservedRunningTime="2026-09-16 18:54:29.427281662 +0000 UTC m=+2307.292185711"
Sep 16 18:54:30.512166 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:30.512131    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:54:38.249172 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:38.249132    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:38.249661 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:38.249187    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:38.262015 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:38.261988    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:38.478450 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:38.478421    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:54:51.226357 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:51.226316    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:53.608112 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.608076    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"]
Sep 16 18:54:53.611464 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.611444    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.614148 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.614127    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0583fa89-kserve-self-signed-certs\""
Sep 16 18:54:53.622984 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.622956    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"]
Sep 16 18:54:53.654028 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.653996    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:53.683355 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.683323    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-tls-certs\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.683518 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.683394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-home\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.683518 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.683416    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-tmp-dir\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.683518 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.683498    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-model-cache\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.683684 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.683525    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-dshm\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.683684 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.683641    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b6b5\" (UniqueName: \"kubernetes.io/projected/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-kube-api-access-8b6b5\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.785304 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.785263    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-dshm\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.785504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.785367    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b6b5\" (UniqueName: \"kubernetes.io/projected/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-kube-api-access-8b6b5\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.785504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.785406    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-tls-certs\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.785504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.785500    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-home\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.785725 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.785526    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-tmp-dir\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.785725 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.785624    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-model-cache\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.786042 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.786010    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-home\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.786159 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.786065    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-model-cache\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.786159 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.786111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-tmp-dir\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.787748 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.787722    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-dshm\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.788027 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.788009    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-tls-certs\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.794758 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.794733    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b6b5\" (UniqueName: \"kubernetes.io/projected/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-kube-api-access-8b6b5\") pod \"e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") " pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:53.925402 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:53.925308    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:54:54.201379 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:54.201339    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"]
Sep 16 18:54:54.202126 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:54:54.202097    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ccbb4d3_7e35_451c_9b26_f67cf3f00dc3.slice/crio-7a06849e673476e39a18e85c3fb7c53abee915157c9638188d1000ac9c256e79 WatchSource:0}: Error finding container 7a06849e673476e39a18e85c3fb7c53abee915157c9638188d1000ac9c256e79: Status 404 returned error can't find the container with id 7a06849e673476e39a18e85c3fb7c53abee915157c9638188d1000ac9c256e79
Sep 16 18:54:54.564513 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:54.564427    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8" event={"ID":"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3","Type":"ContainerStarted","Data":"cb2734f2366bb23a3977141ecc83094f42e009d205c24ca23405d31a38286fc1"}
Sep 16 18:54:54.564513 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:54.564468    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8" event={"ID":"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3","Type":"ContainerStarted","Data":"7a06849e673476e39a18e85c3fb7c53abee915157c9638188d1000ac9c256e79"}
Sep 16 18:54:54.598771 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:54.598718    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8" podStartSLOduration=1.598702863 podStartE2EDuration="1.598702863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:54.598333954 +0000 UTC m=+2332.463238004" watchObservedRunningTime="2026-09-16 18:54:54.598702863 +0000 UTC m=+2332.463606911"
Sep 16 18:54:56.519269 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:54:56.519183    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:55:03.259762 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:03.259733    2562 scope.go:117] "RemoveContainer" containerID="4a1776c5a293786f638534f8e258fd3a28002fc6fd918420c1c323393de79b96"
Sep 16 18:55:03.269522 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:03.269501    2562 scope.go:117] "RemoveContainer" containerID="c408fae62a6427f2ea6c6c4336bebd51884ae8d3689e4402e45d8a3f966739e3"
Sep 16 18:55:03.926161 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:03.926128    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:55:03.926365 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:03.926177    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:55:03.938721 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:03.938691    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:55:04.636321 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:04.636287    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:55:34.543014 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:34.542942    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:40.211689 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:40.211639    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"]
Sep 16 18:55:40.232976 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:40.232940    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:40.723852 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:40.723818    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"]
Sep 16 18:55:40.747177 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:40.747127    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:43.406706 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:43.406663    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:55:53.517334 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:53.517292    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:55:55.231427 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.231383    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt" podUID="4f73ff3d-accf-402e-b781-bb82d2dc6a55" containerName="main" containerID="cri-o://11fa4c54eb9aee449201d4c06107f8cfc77f0f4498a4a46de426bd07442baaa2" gracePeriod=45
Sep 16 18:55:55.495623 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.495557    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:55:55.652557 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.652501    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-dshm\") pod \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") "
Sep 16 18:55:55.652764 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.652589    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdb9f\" (UniqueName: \"kubernetes.io/projected/4f73ff3d-accf-402e-b781-bb82d2dc6a55-kube-api-access-jdb9f\") pod \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") "
Sep 16 18:55:55.652764 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.652624    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-model-cache\") pod \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") "
Sep 16 18:55:55.652764 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.652640    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-tmp-dir\") pod \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") "
Sep 16 18:55:55.652764 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.652689    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f73ff3d-accf-402e-b781-bb82d2dc6a55-tls-certs\") pod \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") "
Sep 16 18:55:55.652764 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.652741    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-home\") pod \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\" (UID: \"4f73ff3d-accf-402e-b781-bb82d2dc6a55\") "
Sep 16 18:55:55.653056 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.652927    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f73ff3d-accf-402e-b781-bb82d2dc6a55" (UID: "4f73ff3d-accf-402e-b781-bb82d2dc6a55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:55.653141 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.653090    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:55:55.653204 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.653157    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-home" (OuterVolumeSpecName: "home") pod "4f73ff3d-accf-402e-b781-bb82d2dc6a55" (UID: "4f73ff3d-accf-402e-b781-bb82d2dc6a55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:55.653204 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.653173    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f73ff3d-accf-402e-b781-bb82d2dc6a55" (UID: "4f73ff3d-accf-402e-b781-bb82d2dc6a55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:55.655016 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.654991    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f73ff3d-accf-402e-b781-bb82d2dc6a55-kube-api-access-jdb9f" (OuterVolumeSpecName: "kube-api-access-jdb9f") pod "4f73ff3d-accf-402e-b781-bb82d2dc6a55" (UID: "4f73ff3d-accf-402e-b781-bb82d2dc6a55"). InnerVolumeSpecName "kube-api-access-jdb9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:55.655144 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.655039    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f73ff3d-accf-402e-b781-bb82d2dc6a55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f73ff3d-accf-402e-b781-bb82d2dc6a55" (UID: "4f73ff3d-accf-402e-b781-bb82d2dc6a55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:55.655144 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.655120    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-dshm" (OuterVolumeSpecName: "dshm") pod "4f73ff3d-accf-402e-b781-bb82d2dc6a55" (UID: "4f73ff3d-accf-402e-b781-bb82d2dc6a55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:55.743420 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.743376    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8" podUID="3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3" containerName="main" containerID="cri-o://cb2734f2366bb23a3977141ecc83094f42e009d205c24ca23405d31a38286fc1" gracePeriod=45
Sep 16 18:55:55.754372 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.754336    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:55:55.754504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.754380    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdb9f\" (UniqueName: \"kubernetes.io/projected/4f73ff3d-accf-402e-b781-bb82d2dc6a55-kube-api-access-jdb9f\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:55:55.754504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.754396    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:55:55.754504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.754411    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f73ff3d-accf-402e-b781-bb82d2dc6a55-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:55:55.754504 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.754424    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f73ff3d-accf-402e-b781-bb82d2dc6a55-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:55:55.927628 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.927591    2562 generic.go:356] "Generic (PLEG): container finished" podID="4f73ff3d-accf-402e-b781-bb82d2dc6a55" containerID="11fa4c54eb9aee449201d4c06107f8cfc77f0f4498a4a46de426bd07442baaa2" exitCode=0
Sep 16 18:55:55.927628 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.927627    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"
Sep 16 18:55:55.927870 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.927671    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt" event={"ID":"4f73ff3d-accf-402e-b781-bb82d2dc6a55","Type":"ContainerDied","Data":"11fa4c54eb9aee449201d4c06107f8cfc77f0f4498a4a46de426bd07442baaa2"}
Sep 16 18:55:55.927870 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.927721    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt" event={"ID":"4f73ff3d-accf-402e-b781-bb82d2dc6a55","Type":"ContainerDied","Data":"d8521efa524e2f0a460c9953b55aabe0f64fe027afc6a6b7ef6af1f274174416"}
Sep 16 18:55:55.927870 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.927745    2562 scope.go:117] "RemoveContainer" containerID="11fa4c54eb9aee449201d4c06107f8cfc77f0f4498a4a46de426bd07442baaa2"
Sep 16 18:55:55.930114 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.930029    2562 generic.go:356] "Generic (PLEG): container finished" podID="3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3" containerID="cb2734f2366bb23a3977141ecc83094f42e009d205c24ca23405d31a38286fc1" exitCode=0
Sep 16 18:55:55.930114 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.930060    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8" event={"ID":"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3","Type":"ContainerDied","Data":"cb2734f2366bb23a3977141ecc83094f42e009d205c24ca23405d31a38286fc1"}
Sep 16 18:55:55.947806 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.947777    2562 scope.go:117] "RemoveContainer" containerID="11fa4c54eb9aee449201d4c06107f8cfc77f0f4498a4a46de426bd07442baaa2"
Sep 16 18:55:55.948256 ip-10-0-142-48 kubenswrapper[2562]: E0916 18:55:55.948224    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11fa4c54eb9aee449201d4c06107f8cfc77f0f4498a4a46de426bd07442baaa2\": container with ID starting with 11fa4c54eb9aee449201d4c06107f8cfc77f0f4498a4a46de426bd07442baaa2 not found: ID does not exist" containerID="11fa4c54eb9aee449201d4c06107f8cfc77f0f4498a4a46de426bd07442baaa2"
Sep 16 18:55:55.948364 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.948264    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11fa4c54eb9aee449201d4c06107f8cfc77f0f4498a4a46de426bd07442baaa2"} err="failed to get container status \"11fa4c54eb9aee449201d4c06107f8cfc77f0f4498a4a46de426bd07442baaa2\": rpc error: code = NotFound desc = could not find container \"11fa4c54eb9aee449201d4c06107f8cfc77f0f4498a4a46de426bd07442baaa2\": container with ID starting with 11fa4c54eb9aee449201d4c06107f8cfc77f0f4498a4a46de426bd07442baaa2 not found: ID does not exist"
Sep 16 18:55:55.966663 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.966633    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"]
Sep 16 18:55:55.970390 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.970365    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0583fa89-kserve-5c4c6f4bc9-8w9xt"]
Sep 16 18:55:55.993604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:55.993539    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:55:56.158938 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.158849    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-model-cache\") pod \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") "
Sep 16 18:55:56.159075 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.158939    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-dshm\") pod \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") "
Sep 16 18:55:56.159075 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.158966    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-home\") pod \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") "
Sep 16 18:55:56.159075 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.158996    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b6b5\" (UniqueName: \"kubernetes.io/projected/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-kube-api-access-8b6b5\") pod \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") "
Sep 16 18:55:56.159075 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.159053    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-tls-certs\") pod \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") "
Sep 16 18:55:56.159240 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.159117    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-tmp-dir\") pod \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\" (UID: \"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3\") "
Sep 16 18:55:56.159240 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.159193    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3" (UID: "3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:56.159387 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.159372    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-model-cache\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:55:56.159463 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.159374    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3" (UID: "3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:56.159532 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.159514    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-home" (OuterVolumeSpecName: "home") pod "3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3" (UID: "3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:56.161176 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.161154    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-dshm" (OuterVolumeSpecName: "dshm") pod "3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3" (UID: "3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:56.161280 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.161211    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3" (UID: "3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:56.161280 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.161224    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-kube-api-access-8b6b5" (OuterVolumeSpecName: "kube-api-access-8b6b5") pod "3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3" (UID: "3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3"). InnerVolumeSpecName "kube-api-access-8b6b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:56.260634 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.260595    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-dshm\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:55:56.260634 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.260629    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-home\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:55:56.261036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.260643    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b6b5\" (UniqueName: \"kubernetes.io/projected/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-kube-api-access-8b6b5\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:55:56.261036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.260657    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-tls-certs\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:55:56.261036 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.260668    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3-tmp-dir\") on node \"ip-10-0-142-48.ec2.internal\" DevicePath \"\""
Sep 16 18:55:56.551146 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.551112    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f73ff3d-accf-402e-b781-bb82d2dc6a55" path="/var/lib/kubelet/pods/4f73ff3d-accf-402e-b781-bb82d2dc6a55/volumes"
Sep 16 18:55:56.937875 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.937790    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"
Sep 16 18:55:56.937875 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.937824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8" event={"ID":"3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3","Type":"ContainerDied","Data":"7a06849e673476e39a18e85c3fb7c53abee915157c9638188d1000ac9c256e79"}
Sep 16 18:55:56.937875 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.937871    2562 scope.go:117] "RemoveContainer" containerID="cb2734f2366bb23a3977141ecc83094f42e009d205c24ca23405d31a38286fc1"
Sep 16 18:55:56.956521 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.956492    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"]
Sep 16 18:55:56.961330 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:56.961305    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0583fa89-kserve-9bcdf8c9f-bqjz8"]
Sep 16 18:55:58.552306 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:58.552274    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3" path="/var/lib/kubelet/pods/3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3/volumes"
Sep 16 18:55:59.723903 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:55:59.723864    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:56:02.789406 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:02.789374    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:56:02.793381 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:02.793350    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:56:02.796898 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:02.796881    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:56:02.800059 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:02.800038    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4wwpr_6e394908-d4ac-40f4-b8ad-232904cc2d64/ovn-acl-logging/0.log"
Sep 16 18:56:03.297396 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:03.297356    2562 scope.go:117] "RemoveContainer" containerID="7f1cf7c6ffa61a9937034cde9516b71e164935cb846b0a6ab396269dcab18b50"
Sep 16 18:56:03.306259 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:03.306236    2562 scope.go:117] "RemoveContainer" containerID="0164efb579aebd1a9ac277d288dbf3cfd1021e9814e86a686d533552049ffb96"
Sep 16 18:56:03.314604 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:03.314581    2562 scope.go:117] "RemoveContainer" containerID="85f58196e3590e58d1e9d044701b0e12a460c795efe6ae48d71b347c88ecdd45"
Sep 16 18:56:07.410768 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:07.410730    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:56:14.579137 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:14.579100    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:56:20.473614 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:20.473559    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:56:27.313764 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:27.313677    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-n726w"]
Sep 16 18:56:29.858453 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:29.858415    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66bc94f9f6-zb4w7_885398f2-92cc-41ef-baa7-15dd7bcb2119/authorino/0.log"
Sep 16 18:56:35.355946 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:35.355914    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mmvjk_51297c36-21d1-48af-8f30-df5470ab7218/manager/0.log"
Sep 16 18:56:35.466717 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:35.466683    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-r2lj9_0f0b145a-64ef-4406-ad4b-23d6357b5ec2/manager/0.log"
Sep 16 18:56:36.130673 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:36.130646    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz_9d750944-d535-40d7-b496-1088d364f6fa/copy-manifests/0.log"
Sep 16 18:56:36.141168 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:36.141141    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz_9d750944-d535-40d7-b496-1088d364f6fa/manager/0.log"
Sep 16 18:56:36.952337 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:36.952304    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826405-cm5qt_bece99ba-a6cb-424e-9187-67fd8e616034/cleanup/0.log"
Sep 16 18:56:37.877433 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:37.877397    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66bc94f9f6-zb4w7_885398f2-92cc-41ef-baa7-15dd7bcb2119/authorino/0.log"
Sep 16 18:56:38.002691 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:38.002645    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-chp8t_6d1bf43f-7668-4f7d-93c4-2a0ba5f7744b/manager/0.log"
Sep 16 18:56:38.127865 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:38.127787    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-xx6jl_824ea996-86ae-4433-8d3c-13b9dcfbf498/manager/0.log"
Sep 16 18:56:38.373874 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:38.373838    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-shb2w_0bafd386-d265-4652-b16b-f6a82bbbb02a/manager/0.log"
Sep 16 18:56:38.521271 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:38.521244    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-n726w_7a958dd8-3400-445a-9733-bb5e512c7b44/limitador/0.log"
Sep 16 18:56:38.646446 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:38.646408    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-vfphc_14542f6e-df6c-4aa1-82e1-fc7602e560ab/manager/0.log"
Sep 16 18:56:39.015952 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:39.015920    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799crdmz4_bc9125d2-cfd5-428d-a511-0f7943dc8ff7/istio-proxy/0.log"
Sep 16 18:56:39.238583 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:39.238529    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79f4c98848-bqq9k_2192ff6e-623b-4bd3-8918-4e584915d514/kube-auth-proxy/0.log"
Sep 16 18:56:39.490223 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:39.490194    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-2mcsv_2c8ffe4b-cb09-4d16-9740-7eba4c4d404f/istio-proxy/1.log"
Sep 16 18:56:39.858867 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:39.858778    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c65ff548d-bxgzt_67a74600-f466-4ae4-b4bf-67ab02745509/router/0.log"
Sep 16 18:56:40.932994 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:40.932962    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-nbjbv_6bb70fce-2990-44c6-a6aa-205efd099f99/main/0.log"
Sep 16 18:56:46.646985 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.646944    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pncdd/must-gather-rftp6"]
Sep 16 18:56:46.647407 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.647340    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3" containerName="main"
Sep 16 18:56:46.647407 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.647358    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3" containerName="main"
Sep 16 18:56:46.647407 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.647382    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f73ff3d-accf-402e-b781-bb82d2dc6a55" containerName="main"
Sep 16 18:56:46.647407 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.647388    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f73ff3d-accf-402e-b781-bb82d2dc6a55" containerName="main"
Sep 16 18:56:46.647601 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.647469    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ccbb4d3-7e35-451c-9b26-f67cf3f00dc3" containerName="main"
Sep 16 18:56:46.647601 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.647483    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f73ff3d-accf-402e-b781-bb82d2dc6a55" containerName="main"
Sep 16 18:56:46.650902 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.650880    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pncdd/must-gather-rftp6"
Sep 16 18:56:46.654011 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.653984    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pncdd\"/\"openshift-service-ca.crt\""
Sep 16 18:56:46.654170 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.654145    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pncdd\"/\"default-dockercfg-5668f\""
Sep 16 18:56:46.654220 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.654148    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pncdd\"/\"kube-root-ca.crt\""
Sep 16 18:56:46.659946 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.659912    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pncdd/must-gather-rftp6"]
Sep 16 18:56:46.704568 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.704505    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:46.802444 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.802410    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2db07d0d-5138-4399-983f-c81fe64fe527-must-gather-output\") pod \"must-gather-rftp6\" (UID: \"2db07d0d-5138-4399-983f-c81fe64fe527\") " pod="openshift-must-gather-pncdd/must-gather-rftp6"
Sep 16 18:56:46.802641 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.802483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlc88\" (UniqueName: \"kubernetes.io/projected/2db07d0d-5138-4399-983f-c81fe64fe527-kube-api-access-jlc88\") pod \"must-gather-rftp6\" (UID: \"2db07d0d-5138-4399-983f-c81fe64fe527\") " pod="openshift-must-gather-pncdd/must-gather-rftp6"
Sep 16 18:56:46.816399 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.816371    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:46.903319 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.903226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2db07d0d-5138-4399-983f-c81fe64fe527-must-gather-output\") pod \"must-gather-rftp6\" (UID: \"2db07d0d-5138-4399-983f-c81fe64fe527\") " pod="openshift-must-gather-pncdd/must-gather-rftp6"
Sep 16 18:56:46.903478 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.903321    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlc88\" (UniqueName: \"kubernetes.io/projected/2db07d0d-5138-4399-983f-c81fe64fe527-kube-api-access-jlc88\") pod \"must-gather-rftp6\" (UID: \"2db07d0d-5138-4399-983f-c81fe64fe527\") " pod="openshift-must-gather-pncdd/must-gather-rftp6"
Sep 16 18:56:46.903684 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.903661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2db07d0d-5138-4399-983f-c81fe64fe527-must-gather-output\") pod \"must-gather-rftp6\" (UID: \"2db07d0d-5138-4399-983f-c81fe64fe527\") " pod="openshift-must-gather-pncdd/must-gather-rftp6"
Sep 16 18:56:46.912880 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.912847    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlc88\" (UniqueName: \"kubernetes.io/projected/2db07d0d-5138-4399-983f-c81fe64fe527-kube-api-access-jlc88\") pod \"must-gather-rftp6\" (UID: \"2db07d0d-5138-4399-983f-c81fe64fe527\") " pod="openshift-must-gather-pncdd/must-gather-rftp6"
Sep 16 18:56:46.960329 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:46.960287    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pncdd/must-gather-rftp6"
Sep 16 18:56:47.252114 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:47.252081    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pncdd/must-gather-rftp6"]
Sep 16 18:56:47.255473 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:56:47.255439    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2db07d0d_5138_4399_983f_c81fe64fe527.slice/crio-4c351f66485d8ab0b5a7f04d9d81ef1164d54c527b0022602c915a05f21e7f65 WatchSource:0}: Error finding container 4c351f66485d8ab0b5a7f04d9d81ef1164d54c527b0022602c915a05f21e7f65: Status 404 returned error can't find the container with id 4c351f66485d8ab0b5a7f04d9d81ef1164d54c527b0022602c915a05f21e7f65
Sep 16 18:56:47.257295 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:47.257270    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:56:48.240963 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:48.240914    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pncdd/must-gather-rftp6" event={"ID":"2db07d0d-5138-4399-983f-c81fe64fe527","Type":"ContainerStarted","Data":"4c351f66485d8ab0b5a7f04d9d81ef1164d54c527b0022602c915a05f21e7f65"}
Sep 16 18:56:49.253850 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:49.253800    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pncdd/must-gather-rftp6" event={"ID":"2db07d0d-5138-4399-983f-c81fe64fe527","Type":"ContainerStarted","Data":"e9a9f788964264d305a6a2e2600b6aca8e68d611a9d706094d1d49731faffd25"}
Sep 16 18:56:49.254369 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:49.253864    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pncdd/must-gather-rftp6" event={"ID":"2db07d0d-5138-4399-983f-c81fe64fe527","Type":"ContainerStarted","Data":"d34476856997c2e62b136ca101a2cae74e6218a411d5233f10e4e78001fd8df0"}
Sep 16 18:56:49.272132 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:49.272077    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pncdd/must-gather-rftp6" podStartSLOduration=2.329907748 podStartE2EDuration="3.272057501s" totalImagesPullingTime="942.149753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:49.269870234 +0000 UTC m=+2447.134774283" watchObservedRunningTime="2026-09-16 18:56:49.272057501 +0000 UTC m=+2447.136961548"
Sep 16 18:56:49.952978 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:49.952943    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ks9dh_a30f7edc-614c-48bc-9559-c7683cc316c0/global-pull-secret-syncer/0.log"
Sep 16 18:56:50.021612 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:50.021573    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gmfcs_d01d46a7-a6c3-4054-8e24-0c9db3c3c9d7/konnectivity-agent/0.log"
Sep 16 18:56:50.159842 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:50.159812    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-48.ec2.internal_f512f1af51c00a31a72cfdbb64042241/haproxy/0.log"
Sep 16 18:56:54.854567 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:54.853265    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66bc94f9f6-zb4w7_885398f2-92cc-41ef-baa7-15dd7bcb2119/authorino/0.log"
Sep 16 18:56:54.891569 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:54.891462    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-chp8t_6d1bf43f-7668-4f7d-93c4-2a0ba5f7744b/manager/0.log"
Sep 16 18:56:54.938601 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:54.938567    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-xx6jl_824ea996-86ae-4433-8d3c-13b9dcfbf498/manager/0.log"
Sep 16 18:56:55.026340 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:55.026305    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-shb2w_0bafd386-d265-4652-b16b-f6a82bbbb02a/manager/0.log"
Sep 16 18:56:55.052164 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:55.052135    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-n726w_7a958dd8-3400-445a-9733-bb5e512c7b44/limitador/0.log"
Sep 16 18:56:55.127182 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:55.127039    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-vfphc_14542f6e-df6c-4aa1-82e1-fc7602e560ab/manager/0.log"
Sep 16 18:56:56.799615 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:56.799576    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-frd2q_06266e44-9985-4fdc-af4f-1476cabba0fd/cluster-monitoring-operator/0.log"
Sep 16 18:56:57.081005 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:57.080904    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ftk6p_45df9a5b-cf94-45a3-9447-08d3b3db251f/node-exporter/0.log"
Sep 16 18:56:57.110960 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:57.110910    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ftk6p_45df9a5b-cf94-45a3-9447-08d3b3db251f/kube-rbac-proxy/0.log"
Sep 16 18:56:57.142414 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:57.142379    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ftk6p_45df9a5b-cf94-45a3-9447-08d3b3db251f/init-textfile/0.log"
Sep 16 18:56:57.655005 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:56:57.654916    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-wbngv_58ad7f7a-8727-4c44-8b62-745b724d70fc/prometheus-operator-admission-webhook/0.log"
Sep 16 18:57:00.111687 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.111650    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/1.log"
Sep 16 18:57:00.118316 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.118247    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-mxm76_e9b403df-9c62-4d05-b8cc-2d2e3d94f521/console-operator/2.log"
Sep 16 18:57:00.789903 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.789862    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-d8qn2_fd86c375-6f0b-445f-9df9-e44ebfea9801/download-server/0.log"
Sep 16 18:57:00.833860 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.833807    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"]
Sep 16 18:57:00.839466 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.839431    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:00.855122 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.854807    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"]
Sep 16 18:57:00.870808 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.870340    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30656991-d6dd-4c78-8b62-88db2d7badfa-lib-modules\") pod \"perf-node-gather-daemonset-9c9xg\" (UID: \"30656991-d6dd-4c78-8b62-88db2d7badfa\") " pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:00.870808 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.870401    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30656991-d6dd-4c78-8b62-88db2d7badfa-podres\") pod \"perf-node-gather-daemonset-9c9xg\" (UID: \"30656991-d6dd-4c78-8b62-88db2d7badfa\") " pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:00.870808 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.870502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5tk7\" (UniqueName: \"kubernetes.io/projected/30656991-d6dd-4c78-8b62-88db2d7badfa-kube-api-access-g5tk7\") pod \"perf-node-gather-daemonset-9c9xg\" (UID: \"30656991-d6dd-4c78-8b62-88db2d7badfa\") " pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:00.870808 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.870571    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30656991-d6dd-4c78-8b62-88db2d7badfa-sys\") pod \"perf-node-gather-daemonset-9c9xg\" (UID: \"30656991-d6dd-4c78-8b62-88db2d7badfa\") " pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:00.870808 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.870597    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30656991-d6dd-4c78-8b62-88db2d7badfa-proc\") pod \"perf-node-gather-daemonset-9c9xg\" (UID: \"30656991-d6dd-4c78-8b62-88db2d7badfa\") " pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:00.971988 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.971940    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5tk7\" (UniqueName: \"kubernetes.io/projected/30656991-d6dd-4c78-8b62-88db2d7badfa-kube-api-access-g5tk7\") pod \"perf-node-gather-daemonset-9c9xg\" (UID: \"30656991-d6dd-4c78-8b62-88db2d7badfa\") " pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:00.972183 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.972028    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30656991-d6dd-4c78-8b62-88db2d7badfa-sys\") pod \"perf-node-gather-daemonset-9c9xg\" (UID: \"30656991-d6dd-4c78-8b62-88db2d7badfa\") " pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:00.972183 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.972056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30656991-d6dd-4c78-8b62-88db2d7badfa-proc\") pod \"perf-node-gather-daemonset-9c9xg\" (UID: \"30656991-d6dd-4c78-8b62-88db2d7badfa\") " pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:00.972183 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.972104    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30656991-d6dd-4c78-8b62-88db2d7badfa-lib-modules\") pod \"perf-node-gather-daemonset-9c9xg\" (UID: \"30656991-d6dd-4c78-8b62-88db2d7badfa\") " pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:00.972183 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.972130    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30656991-d6dd-4c78-8b62-88db2d7badfa-podres\") pod \"perf-node-gather-daemonset-9c9xg\" (UID: \"30656991-d6dd-4c78-8b62-88db2d7badfa\") " pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:00.972423 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.972380    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30656991-d6dd-4c78-8b62-88db2d7badfa-podres\") pod \"perf-node-gather-daemonset-9c9xg\" (UID: \"30656991-d6dd-4c78-8b62-88db2d7badfa\") " pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:00.972887 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.972857    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30656991-d6dd-4c78-8b62-88db2d7badfa-sys\") pod \"perf-node-gather-daemonset-9c9xg\" (UID: \"30656991-d6dd-4c78-8b62-88db2d7badfa\") " pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:00.973023 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.972929    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30656991-d6dd-4c78-8b62-88db2d7badfa-proc\") pod \"perf-node-gather-daemonset-9c9xg\" (UID: \"30656991-d6dd-4c78-8b62-88db2d7badfa\") " pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:00.973091 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.973037    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30656991-d6dd-4c78-8b62-88db2d7badfa-lib-modules\") pod \"perf-node-gather-daemonset-9c9xg\" (UID: \"30656991-d6dd-4c78-8b62-88db2d7badfa\") " pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:00.985511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:00.985387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5tk7\" (UniqueName: \"kubernetes.io/projected/30656991-d6dd-4c78-8b62-88db2d7badfa-kube-api-access-g5tk7\") pod \"perf-node-gather-daemonset-9c9xg\" (UID: \"30656991-d6dd-4c78-8b62-88db2d7badfa\") " pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:01.156156 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:01.156046    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:01.487511 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:01.486766    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"]
Sep 16 18:57:01.489417 ip-10-0-142-48 kubenswrapper[2562]: W0916 18:57:01.489371    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod30656991_d6dd_4c78_8b62_88db2d7badfa.slice/crio-e208320dac72983d91a599ed539afbc830fefe031a9381ca92d16698a9aec02e WatchSource:0}: Error finding container e208320dac72983d91a599ed539afbc830fefe031a9381ca92d16698a9aec02e: Status 404 returned error can't find the container with id e208320dac72983d91a599ed539afbc830fefe031a9381ca92d16698a9aec02e
Sep 16 18:57:01.666958 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:01.666915    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-gcvrf_92d1ca76-661a-47d9-bb84-e11ee89825e7/volume-data-source-validator/0.log"
Sep 16 18:57:02.391384 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:02.391337    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg" event={"ID":"30656991-d6dd-4c78-8b62-88db2d7badfa","Type":"ContainerStarted","Data":"6bc813549a686da7f3cf2b95b21dc0c1a090af2e35631c0d3f8f54548eff4791"}
Sep 16 18:57:02.391871 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:02.391412    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg" event={"ID":"30656991-d6dd-4c78-8b62-88db2d7badfa","Type":"ContainerStarted","Data":"e208320dac72983d91a599ed539afbc830fefe031a9381ca92d16698a9aec02e"}
Sep 16 18:57:02.391871 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:02.391456    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:02.415200 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:02.415139    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg" podStartSLOduration=2.415118296 podStartE2EDuration="2.415118296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:02.41258806 +0000 UTC m=+2460.277492109" watchObservedRunningTime="2026-09-16 18:57:02.415118296 +0000 UTC m=+2460.280022343"
Sep 16 18:57:03.097823 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:03.097773    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bcg6p_1b6340b4-d303-4081-b1c6-c12e28cb1a31/dns/0.log"
Sep 16 18:57:03.123488 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:03.123436    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bcg6p_1b6340b4-d303-4081-b1c6-c12e28cb1a31/kube-rbac-proxy/0.log"
Sep 16 18:57:03.223331 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:03.223299    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x5xdc_d5d1838b-753e-4591-b50a-473909c62656/dns-node-resolver/0.log"
Sep 16 18:57:03.874264 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:03.874229    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-65kl9_e04d0991-aae5-4412-850d-6e4685c0124a/node-ca/0.log"
Sep 16 18:57:04.992369 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:04.992327    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799crdmz4_bc9125d2-cfd5-428d-a511-0f7943dc8ff7/istio-proxy/0.log"
Sep 16 18:57:05.366534 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:05.366456    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79f4c98848-bqq9k_2192ff6e-623b-4bd3-8918-4e584915d514/kube-auth-proxy/0.log"
Sep 16 18:57:05.427352 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:05.427317    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-2mcsv_2c8ffe4b-cb09-4d16-9740-7eba4c4d404f/istio-proxy/0.log"
Sep 16 18:57:05.431037 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:05.431008    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-2mcsv_2c8ffe4b-cb09-4d16-9740-7eba4c4d404f/istio-proxy/1.log"
Sep 16 18:57:05.528818 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:05.528786    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c65ff548d-bxgzt_67a74600-f466-4ae4-b4bf-67ab02745509/router/0.log"
Sep 16 18:57:06.553464 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:06.553437    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vnc87_743d0ab1-2506-4fba-951d-80eee5871235/serve-healthcheck-canary/0.log"
Sep 16 18:57:07.093320 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:07.093288    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-wnqkf_4642a843-fe4e-43a4-ab7e-64cc1c66bf10/insights-operator/0.log"
Sep 16 18:57:07.093836 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:07.093815    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-wnqkf_4642a843-fe4e-43a4-ab7e-64cc1c66bf10/insights-operator/1.log"
Sep 16 18:57:07.203537 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:07.203507    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p4k2k_da9b816c-32e3-4d22-8940-806ebf564ffe/kube-rbac-proxy/0.log"
Sep 16 18:57:07.226350 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:07.226313    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p4k2k_da9b816c-32e3-4d22-8940-806ebf564ffe/exporter/0.log"
Sep 16 18:57:07.251050 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:07.251018    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p4k2k_da9b816c-32e3-4d22-8940-806ebf564ffe/extractor/0.log"
Sep 16 18:57:08.410522 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:08.410487    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pncdd/perf-node-gather-daemonset-9c9xg"
Sep 16 18:57:09.892419 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:09.892388    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-mmvjk_51297c36-21d1-48af-8f30-df5470ab7218/manager/0.log"
Sep 16 18:57:10.202009 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:10.201842    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-r2lj9_0f0b145a-64ef-4406-ad4b-23d6357b5ec2/manager/0.log"
Sep 16 18:57:10.483563 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:10.483490    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz_9d750944-d535-40d7-b496-1088d364f6fa/manager/0.log"
Sep 16 18:57:10.510559 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:10.510510    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cfd5c8dfd-vmzlz_9d750944-d535-40d7-b496-1088d364f6fa/copy-manifests/0.log"
Sep 16 18:57:11.413108 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:11.413074    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799crdmz4_bc9125d2-cfd5-428d-a511-0f7943dc8ff7/istio-proxy/0.log"
Sep 16 18:57:11.769681 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:11.769650    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79f4c98848-bqq9k_2192ff6e-623b-4bd3-8918-4e584915d514/kube-auth-proxy/0.log"
Sep 16 18:57:11.828819 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:11.828779    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-2mcsv_2c8ffe4b-cb09-4d16-9740-7eba4c4d404f/istio-proxy/0.log"
Sep 16 18:57:11.832525 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:11.832489    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-2mcsv_2c8ffe4b-cb09-4d16-9740-7eba4c4d404f/istio-proxy/1.log"
Sep 16 18:57:11.970791 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:11.970764    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c65ff548d-bxgzt_67a74600-f466-4ae4-b4bf-67ab02745509/router/0.log"
Sep 16 18:57:13.609598 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:13.609527    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799crdmz4_bc9125d2-cfd5-428d-a511-0f7943dc8ff7/istio-proxy/0.log"
Sep 16 18:57:13.992686 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:13.992651    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79f4c98848-bqq9k_2192ff6e-623b-4bd3-8918-4e584915d514/kube-auth-proxy/0.log"
Sep 16 18:57:14.029851 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:14.029819    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-2mcsv_2c8ffe4b-cb09-4d16-9740-7eba4c4d404f/istio-proxy/0.log"
Sep 16 18:57:14.034060 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:14.034026    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-2mcsv_2c8ffe4b-cb09-4d16-9740-7eba4c4d404f/istio-proxy/1.log"
Sep 16 18:57:14.131536 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:14.131501    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c65ff548d-bxgzt_67a74600-f466-4ae4-b4bf-67ab02745509/router/0.log"
Sep 16 18:57:18.385817 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:18.385722    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-wrll4_4a69f057-7c16-42af-8e69-9be2c06c02c7/kube-storage-version-migrator-operator/1.log"
Sep 16 18:57:18.386660 ip-10-0-142-48 kubenswrapper[2562]: I0916 18:57:18.386642    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-wrll4_4a69f057-7c16-42af-8e69-9be2c06c02c7/kube-storage-version-migrator-operator/0.log"