Sep 11 17:18:38.037290 ip-10-0-138-148 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 17:18:38.037301 ip-10-0-138-148 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 17:18:38.037308 ip-10-0-138-148 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 17:18:38.037467 ip-10-0-138-148 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 17:18:48.161145 ip-10-0-138-148 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 17:18:48.161161 ip-10-0-138-148 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 33b7fe8500a84b5686b53ed2a4c7eb30 --
Sep 11 17:20:28.420693 ip-10-0-138-148 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 17:20:28.815327 ip-10-0-138-148 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:20:28.815327 ip-10-0-138-148 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 17:20:28.815327 ip-10-0-138-148 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:20:28.815327 ip-10-0-138-148 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 17:20:28.815327 ip-10-0-138-148 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:20:28.817172 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.817083    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 17:20:28.821409 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821389    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:20:28.821409 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821404    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:20:28.821409 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821408    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:20:28.821409 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821411    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:20:28.821409 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821414    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821417    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821420    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821435    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821438    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821442    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821448    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821451    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821454    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821457    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821461    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821463    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821466    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821470    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821474    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821480    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821483    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821485    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821488    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:20:28.821603 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821491    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821493    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821496    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821499    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821501    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821504    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821507    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821509    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821512    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821515    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821517    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821520    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821522    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821525    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821527    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821531    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821534    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821536    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821541    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821545    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:20:28.822052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821548    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821551    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821554    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821557    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821560    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821563    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821565    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821569    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821572    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821575    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821577    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821580    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821582    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821585    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821588    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821590    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821593    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821595    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821598    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821601    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:20:28.822588 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821603    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821606    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821609    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821612    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821615    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821619    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821622    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821624    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821627    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821630    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821633    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821635    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821638    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821641    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821644    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821646    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821650    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821652    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821655    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821657    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:20:28.823080 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821661    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821663    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821666    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821669    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821671    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821674    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821676    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821679    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821681    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821684    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821687    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821690    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821693    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821695    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821698    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.821700    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822393    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822400    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822403    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822405    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:20:28.823586 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822409    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822412    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822415    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822418    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822421    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822423    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822426    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822429    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822431    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822434    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822436    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822439    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822442    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822445    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822448    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822450    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822453    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822456    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822459    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:20:28.824065 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822461    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822464    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822466    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822469    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822472    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822475    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822478    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822480    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822483    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822486    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822494    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822497    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822500    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822503    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822505    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822508    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822510    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822513    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822516    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822519    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:20:28.824551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822521    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822524    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822527    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822529    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822532    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822534    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822537    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822540    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822543    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822545    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822548    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822551    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822554    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822556    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822559    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822562    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822566    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822568    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822571    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822574    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:20:28.825086 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822576    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822579    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822582    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822585    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822587    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822590    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822593    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822595    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822598    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822601    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822604    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822606    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822609    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822611    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822614    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822617    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822619    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822622    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822624    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822627    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:20:28.825605 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822630    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822632    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822635    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822637    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822641    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822643    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822646    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822649    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822651    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822654    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822658    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822662    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822665    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822669    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822676    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.822680    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823048    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823053    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823056    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:20:28.826101 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823059    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823062    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823065    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823067    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823070    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823073    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823075    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823078    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823081    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823084    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823086    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823089    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823092    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823095    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823098    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823101    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823103    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823106    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823109    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:20:28.826573 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823111    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823114    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823116    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823119    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823122    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823127    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823130    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823133    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823138    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823141    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823144    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823146    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823149    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823152    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823155    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823158    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823161    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823163    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823166    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823169    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:20:28.827032 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823172    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823174    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823177    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823180    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823183    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823186    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823188    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823191    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823194    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823197    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823200    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823203    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823206    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823209    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823212    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823216    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823219    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823222    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823225    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:20:28.827528 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823228    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823232    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823234    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823239    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823241    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823244    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823247    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823249    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823252    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823254    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823257    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823259    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823262    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823264    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823267    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823270    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823272    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823275    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823277    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823280    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:20:28.828008 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823282    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823286    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823289    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823291    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823294    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823296    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823300    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823303    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823305    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823308    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823310    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823313    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823315    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823319    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823322    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823325    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823328    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.823330    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823418    2560 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823452    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823458    2560 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 17:20:28.828509 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823462    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823466    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823470    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823474    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823478    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823481    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823484    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823487    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823491    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823494    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823496    2560 flags.go:64] FLAG: --cgroup-root=""
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823500    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823503    2560 flags.go:64] FLAG: --client-ca-file=""
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823507    2560 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823510    2560 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823514    2560 flags.go:64] FLAG: --cluster-domain=""
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823517    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823521    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823525    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823528    2560 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823532    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823535    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823538    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 17:20:28.829111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823541    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823546    2560 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823549    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823552    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823555    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823558    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823562    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823565    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823568    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823571    2560 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823574    2560 flags.go:64] FLAG: --enable-server="true"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823576    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823581    2560 flags.go:64] FLAG: --event-burst="100"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823584    2560 flags.go:64] FLAG: --event-qps="50"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823587    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823590    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823593    2560 flags.go:64] FLAG: --eviction-hard=""
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823597    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823600    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823603    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823606    2560 flags.go:64] FLAG: --eviction-soft=""
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823609    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823612    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823615    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823618    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 17:20:28.829687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823621    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823624    2560 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823627    2560 flags.go:64] FLAG: --feature-gates=""
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823631    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823634    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823637    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823640    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823643    2560 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823646    2560 flags.go:64] FLAG: --help="false"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823649    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-138-148.ec2.internal"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823652    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823655    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823658    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823661    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823664    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823667    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823670    2560 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823673    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823675    2560 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823678    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823681    2560 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823684    2560 flags.go:64] FLAG: --kube-reserved=""
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823687    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823689    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 17:20:28.830329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823692    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823695    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823698    2560 flags.go:64] FLAG: --lock-file=""
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823700    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823703    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823706    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823712    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823715    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823718    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823721    2560 flags.go:64] FLAG: --logging-format="text"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823724    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823727    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823730    2560 flags.go:64] FLAG: --manifest-url=""
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823733    2560 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823737    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823740    2560 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823744    2560 flags.go:64] FLAG: --max-pods="110"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823748    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823751    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823754    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823757    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823760    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823762    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823765    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823772    2560 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 17:20:28.830936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823775    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823778    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823781    2560 flags.go:64] FLAG: --pod-cidr=""
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823784    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823789    2560 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823792    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823795    2560 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823798    2560 flags.go:64] FLAG: --port="10250"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823801    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823804    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-085f9476eb028b535"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823807    2560 flags.go:64] FLAG: --qos-reserved=""
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823810    2560 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823813    2560 flags.go:64] FLAG: --register-node="true"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823816    2560 flags.go:64] FLAG: --register-with-taints=""
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823820    2560 flags.go:64] FLAG: --registry-burst="10"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823828    2560 flags.go:64] FLAG: --registry-qps="5"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823831    2560 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823834    2560 flags.go:64] FLAG: --reserved-memory=""
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823837    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823841    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823844    2560 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823846    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823849    2560 flags.go:64] FLAG: --runonce="false"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823852    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823855    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 17:20:28.831560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823858    2560 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823861    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823864    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823867    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823870    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823873    2560 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823876    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823879    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823882    2560 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823885    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823888    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823891    2560 flags.go:64] FLAG: --system-cgroups=""
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823893    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823898    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823901    2560 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823904    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823908    2560 flags.go:64] FLAG: --tls-min-version=""
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823911    2560 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823913    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823916    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823919    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823922    2560 flags.go:64] FLAG: --v="2"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823926    2560 flags.go:64] FLAG: --version="false"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823930    2560 flags.go:64] FLAG: --vmodule=""
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823934    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 17:20:28.832211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.823937    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824050    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824054    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824057    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824060    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824063    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824066    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824069    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824072    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824075    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824079    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824082    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824085    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824087    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824090    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824093    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824095    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824098    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824100    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824103    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:20:28.832827 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824105    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824110    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824112    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824115    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824117    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824120    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824123    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824125    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824128    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824130    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824133    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824135    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824138    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824141    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824144    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824146    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824150    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824153    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824156    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824158    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:20:28.833314 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824161    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824164    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824167    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824169    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824172    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824175    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824177    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824180    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824182    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824185    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824188    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824190    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824193    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824197    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824199    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824202    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824204    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824207    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824210    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824213    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:20:28.833879 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824215    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824218    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824220    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824223    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824226    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824229    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824231    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824234    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824239    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824243    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824246    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824248    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824251    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824254    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824256    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824259    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824262    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824264    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824267    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:20:28.834412 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824271    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824274    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824277    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824281    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824283    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824286    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824290    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824293    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824296    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824300    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824303    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824307    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824309    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824312    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824315    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824317    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824320    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824323    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824326    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:20:28.834886 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824328    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:20:28.835345 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.824331    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:20:28.835345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.824825    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:20:28.835345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.831640    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 17:20:28.835345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.831655    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 17:20:28.835345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.832416    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 17:20:28.835345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.832423    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 17:20:28.835345 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832497    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:20:28.835345 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832502    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:20:28.835345 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832505    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:20:28.835345 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832508    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:20:28.835345 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832510    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:20:28.835345 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832513    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:20:28.835345 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832516    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:20:28.835345 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832519    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:20:28.835345 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832522    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832525    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832528    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832531    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832533    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832536    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832539    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832541    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832544    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832547    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832549    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832552    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832554    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832557    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832560    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832562    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832566    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832568    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832571    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832574    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:20:28.835731 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832577    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832580    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832591    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832594    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832596    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832599    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832602    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832605    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832608    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832610    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832613    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832615    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832618    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832621    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832624    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832626    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832629    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832632    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832634    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832637    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:20:28.836209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832639    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832642    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832645    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832647    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832650    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832653    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832659    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832662    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832665    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832669    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832673    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832676    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832679    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832682    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832685    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832688    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832691    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832693    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832696    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:20:28.836713 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832699    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832703    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832706    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832709    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832711    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832714    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832716    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832719    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832722    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832725    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832729    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832732    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832734    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832737    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832740    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832743    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832745    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832748    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832751    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:20:28.837185 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832753    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:20:28.837691 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832756    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:20:28.837691 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832758    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:20:28.837691 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832762    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:20:28.837691 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832765    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:20:28.837691 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832768    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:20:28.837691 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832771    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:20:28.837691 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832774    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:20:28.837691 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832776    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:20:28.837691 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832779    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:20:28.837691 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832782    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:20:28.837691 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832784    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:20:28.837691 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832786    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:20:28.837691 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.832791    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:20:28.837691 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832896    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:20:28.837691 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832900    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832903    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832906    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832908    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832911    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832914    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832916    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832919    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832921    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832924    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832926    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832929    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832931    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832934    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832937    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832939    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832942    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832944    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832947    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832950    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:20:28.838068 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832952    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832955    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832958    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832960    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832963    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832966    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832969    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832971    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832974    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832976    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832979    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832982    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832984    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832987    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832989    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832992    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832994    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.832997    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833000    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833002    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:20:28.838566 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833005    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833007    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833009    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833012    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833014    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833017    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833019    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833022    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833024    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833027    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833030    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833033    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833035    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833038    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833041    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833045    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833048    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833051    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833053    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833056    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:20:28.839052 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833058    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833061    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833063    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833067    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833070    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833073    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833076    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833079    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833081    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833084    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833087    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833089    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833092    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833094    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833097    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833099    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833102    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833105    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833108    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:20:28.839551 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833111    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833114    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833116    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833119    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833122    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833124    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833127    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833130    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833133    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833136    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833138    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833140    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833143    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833145    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833148    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833150    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833153    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833155    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:20:28.840006 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:28.833158    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:20:28.840449 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.833162    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:20:28.840449 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.833250    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 17:20:28.840449 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.835049    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 17:20:28.840449 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.836079    2560 server.go:1019] "Starting client certificate rotation"
Sep 11 17:20:28.840449 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.836178    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:20:28.840449 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.836228    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:20:28.840449 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.838622    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:20:28.841533 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.841516    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:20:28.856493 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.856472    2560 log.go:25] "Validated CRI v1 runtime API"
Sep 11 17:20:28.859513 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.859495    2560 log.go:25] "Validated CRI v1 image API"
Sep 11 17:20:28.860283 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.860268    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 17:20:28.863208 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.863189    2560 fs.go:135] Filesystem UUIDs: map[12897976-eac4-4f29-807c-2fda2f5757d5:/dev/nvme0n1p4 7844b49e-6163-460f-b6b7-ccb664c552f7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 11 17:20:28.863259 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.863208    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 17:20:28.865453 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.865432    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:20:28.869406 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.869257    2560 manager.go:216] Machine: {Timestamp:2026-09-11 17:20:28.867885946 +0000 UTC m=+0.323447319 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096105 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eebbf00d2d0ca8083da5f2ebad88d SystemUUID:ec2eebbf-00d2-d0ca-8083-da5f2ebad88d BootID:33b7fe85-00a8-4b56-86b5-3ed2a4c7eb30 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:95:c7:cf:78:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:95:c7:cf:78:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:aa:d2:92:87:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 17:20:28.869406 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.869403    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 17:20:28.869536 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.869481    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 17:20:28.872519 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.872495    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 17:20:28.872694 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.872521    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-148.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 17:20:28.873293 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.873281    2560 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 17:20:28.873348 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.873298    2560 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 17:20:28.873348 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.873310    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:20:28.874109 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.874097    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:20:28.874862 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.874851    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 17:20:28.877618 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.877604    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:20:28.877747 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.877736    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 17:20:28.877802 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.877774    2560 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 17:20:28.877802 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.877789    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 17:20:28.877891 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.877811    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 17:20:28.877891 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.877827    2560 kubelet.go:398] "Adding apiserver pod source"
Sep 11 17:20:28.877891 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.877841    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 17:20:28.878222 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.878208    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:20:28.878338 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.878233    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:20:28.880402 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.880382    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 17:20:28.882118 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.882104    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 17:20:28.882173 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.882134    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 17:20:28.883491 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.883479    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 17:20:28.883541 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.883499    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 17:20:28.883541 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.883507    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 17:20:28.883541 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.883512    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 17:20:28.883541 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.883521    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 17:20:28.883541 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.883526    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 17:20:28.883541 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.883540    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 17:20:28.883695 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.883545    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 17:20:28.883695 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.883552    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 17:20:28.883695 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.883557    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 17:20:28.883695 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.883570    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 17:20:28.883695 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.883575    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 17:20:28.884376 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.884349    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 17:20:28.884376 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.884360    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 17:20:28.887312 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:28.887289    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 17:20:28.887387 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.887351    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-148.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 17:20:28.887455 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:28.887440    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 17:20:28.888283 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.888271    2560 server.go:1268] "Started kubelet"
Sep 11 17:20:28.888770 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.888748    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 17:20:28.888851 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.888824    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 17:20:28.888914 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.888877    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 17:20:28.889351 ip-10-0-138-148 systemd[1]: Started Kubernetes Kubelet.
Sep 11 17:20:28.890888 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.890869    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 17:20:28.891332 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.891308    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mzhqn"
Sep 11 17:20:28.892590 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.892565    2560 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 17:20:28.895291 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.895258    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mzhqn"
Sep 11 17:20:28.895597 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:28.895578    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 17:20:28.896013 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.895995    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 17:20:28.896633 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.896617    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 17:20:28.897514 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:28.897495    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-148.ec2.internal\" not found"
Sep 11 17:20:28.897976 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.897834    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 17:20:28.897976 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.897852    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 17:20:28.897976 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.897894    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 17:20:28.897976 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.897968    2560 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 17:20:28.897976 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.897978    2560 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 17:20:28.898356 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.898339    2560 factory.go:55] Registering systemd factory
Sep 11 17:20:28.898429 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.898359    2560 factory.go:223] Registration of the systemd container factory successfully
Sep 11 17:20:28.898495 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:28.897553    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-148.ec2.internal.18d453ece4c23df6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-148.ec2.internal,UID:ip-10-0-138-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-148.ec2.internal,},FirstTimestamp:2026-09-11 17:20:28.888251894 +0000 UTC m=+0.343813267,LastTimestamp:2026-09-11 17:20:28.888251894 +0000 UTC m=+0.343813267,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-148.ec2.internal,}"
Sep 11 17:20:28.898657 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.898593    2560 factory.go:153] Registering CRI-O factory
Sep 11 17:20:28.898657 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.898610    2560 factory.go:223] Registration of the crio container factory successfully
Sep 11 17:20:28.898772 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.898664    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 17:20:28.898772 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.898683    2560 factory.go:103] Registering Raw factory
Sep 11 17:20:28.898772 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.898697    2560 manager.go:1196] Started watching for new ooms in manager
Sep 11 17:20:28.899481 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.899467    2560 manager.go:349] Starting recovery of all containers
Sep 11 17:20:28.907195 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.907024    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:20:28.909072 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.909060    2560 manager.go:354] Recovery completed
Sep 11 17:20:28.909357 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:28.909335    2560 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-148.ec2.internal\" not found" node="ip-10-0-138-148.ec2.internal"
Sep 11 17:20:28.913093 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.913081    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:20:28.915189 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.915167    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:20:28.915265 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.915205    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:20:28.915265 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.915219    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:20:28.915831 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.915815    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 17:20:28.915831 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.915829    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 17:20:28.915977 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.915844    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:20:28.915977 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.915849    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:20:28.916651 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.916637    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:20:28.916718 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.916667    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:20:28.916718 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.916681    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:20:28.917074 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.917060    2560 policy_none.go:49] "None policy: Start"
Sep 11 17:20:28.917114 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.917080    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 17:20:28.917114 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.917093    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 17:20:28.919585 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.919573    2560 policy_none.go:47] "Start"
Sep 11 17:20:28.946066 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.946048    2560 manager.go:337] "Starting Device Plugin manager"
Sep 11 17:20:28.956387 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:28.946095    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 17:20:28.956387 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.946108    2560 server.go:87] "Starting device plugin registration server"
Sep 11 17:20:28.956387 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.946506    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 17:20:28.956387 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.946520    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 17:20:28.956387 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.946611    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 17:20:28.956387 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.946723    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 17:20:28.956387 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:28.946732    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 17:20:28.956387 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:28.947337    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 17:20:28.956387 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:28.947402    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-148.ec2.internal\" not found"
Sep 11 17:20:29.019886 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.019848    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 17:20:29.021014 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.020993    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 17:20:29.021093 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.021068    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 17:20:29.021135 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.021098    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 17:20:29.021179 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:29.021147    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 17:20:29.023459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.023438    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:20:29.047264 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.047235    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:20:29.048212 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.048194    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:20:29.048296 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.048228    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:20:29.048296 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.048243    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:20:29.048296 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.048272    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.053967 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.053953    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.054021 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:29.053974    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-148.ec2.internal\": node \"ip-10-0-138-148.ec2.internal\" not found"
Sep 11 17:20:29.069008 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:29.068935    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-148.ec2.internal\" not found"
Sep 11 17:20:29.122093 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.122068    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-148.ec2.internal"]
Sep 11 17:20:29.122207 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.122137    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:20:29.123666 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.123630    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:20:29.123666 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.123666    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:20:29.123829 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.123680    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:20:29.124897 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.124880    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:20:29.125005 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.124990    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.125055 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.125021    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:20:29.125612 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.125597    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:20:29.125661 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.125622    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:20:29.125661 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.125632    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:20:29.126075 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.126058    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:20:29.126152 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.126086    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:20:29.126152 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.126097    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:20:29.127201 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.127187    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.127247 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.127212    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:20:29.127858 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.127843    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:20:29.127941 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.127871    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:20:29.127941 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.127884    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:20:29.141909 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:29.141886    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-148.ec2.internal\" not found" node="ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.146164 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:29.146150    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-148.ec2.internal\" not found" node="ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.169204 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:29.169187    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-148.ec2.internal\" not found"
Sep 11 17:20:29.199941 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.199924    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40235f875029f0595008c74b069417b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal\" (UID: \"40235f875029f0595008c74b069417b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.200013 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.199952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40235f875029f0595008c74b069417b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal\" (UID: \"40235f875029f0595008c74b069417b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.200013 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.199971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcffbb3555202a1c58f0c3380c4a7532-config\") pod \"kube-apiserver-proxy-ip-10-0-138-148.ec2.internal\" (UID: \"fcffbb3555202a1c58f0c3380c4a7532\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.269680 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:29.269648    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-148.ec2.internal\" not found"
Sep 11 17:20:29.301060 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.301037    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40235f875029f0595008c74b069417b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal\" (UID: \"40235f875029f0595008c74b069417b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.301139 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.301067    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40235f875029f0595008c74b069417b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal\" (UID: \"40235f875029f0595008c74b069417b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.301139 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.301084    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcffbb3555202a1c58f0c3380c4a7532-config\") pod \"kube-apiserver-proxy-ip-10-0-138-148.ec2.internal\" (UID: \"fcffbb3555202a1c58f0c3380c4a7532\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.301196 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.301144    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40235f875029f0595008c74b069417b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal\" (UID: \"40235f875029f0595008c74b069417b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.301196 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.301154    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcffbb3555202a1c58f0c3380c4a7532-config\") pod \"kube-apiserver-proxy-ip-10-0-138-148.ec2.internal\" (UID: \"fcffbb3555202a1c58f0c3380c4a7532\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.301196 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.301149    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40235f875029f0595008c74b069417b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal\" (UID: \"40235f875029f0595008c74b069417b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.370424 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:29.370309    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-148.ec2.internal\" not found"
Sep 11 17:20:29.443941 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.443900    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.448523 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:29.448503    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-148.ec2.internal"
Sep 11 17:20:29.471051 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:29.471023    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-148.ec2.internal\" not found"
Sep 11 17:20:30.594661 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:30.594609    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-148.ec2.internal\" not found"
Sep 11 17:20:30.695125 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:30.695060    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-148.ec2.internal\" not found"
Sep 11 17:20:30.725155 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.725131    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:20:30.821151 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.821125    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal"
Sep 11 17:20:30.833176 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.833157    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:20:30.835237 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.835223    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-148.ec2.internal"
Sep 11 17:20:30.841565 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.841552    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:20:30.858725 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.858706    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 17:20:30.858886 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.858858    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 17:20:30.858940 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.858863    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 17:20:30.858940 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.858865    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 17:20:30.902003 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.901979    2560 apiserver.go:52] "Watching apiserver"
Sep 11 17:20:30.917231 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.917206    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 17:20:30.917587 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.917568    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-fmkwp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal","openshift-multus/multus-additional-cni-plugins-24d5m","openshift-multus/multus-dv8fv","openshift-multus/network-metrics-daemon-8zghj","kube-system/konnectivity-agent-5dl9t","openshift-cluster-node-tuning-operator/tuned-xlx9t","openshift-image-registry/node-ca-ttqn4","openshift-network-diagnostics/network-check-target-4lcjd","openshift-network-operator/iptables-alerter-b7t5c","openshift-ovn-kubernetes/ovnkube-node-mdw6t","kube-system/global-pull-secret-syncer-b8pb5","kube-system/kube-apiserver-proxy-ip-10-0-138-148.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"]
Sep 11 17:20:30.918836 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.918820    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fmkwp"
Sep 11 17:20:30.919314 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.919289    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 17:20:30.922149 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.921886    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:30.922149 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.921898    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 17:20:30.922149 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.922076    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2fb2w\""
Sep 11 17:20:30.922149 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.922080    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 17:15:29 +0000 UTC" deadline="2028-06-22 16:51:18.797581946 +0000 UTC"
Sep 11 17:20:30.922149 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.922121    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15599h30m47.875464604s"
Sep 11 17:20:30.922437 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.922185    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 17:20:30.923126 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.923110    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:30.924264 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.924119    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 17:20:30.925138 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.924930    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6msk6\""
Sep 11 17:20:30.925138 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.925095    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 17:20:30.925288 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.925274    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 17:20:30.925450 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.925431    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:30.925564 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:30.925509    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:30.925628 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.925581    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 17:20:30.925672 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.925627    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 17:20:30.925706 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.925694    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 17:20:30.925752 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.925710    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nb4pr\""
Sep 11 17:20:30.926511 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.926492    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5dl9t"
Sep 11 17:20:30.927184 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.927160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ccc7a17-e4f2-48ac-a2b1-308eda0170f4-hosts-file\") pod \"node-resolver-fmkwp\" (UID: \"8ccc7a17-e4f2-48ac-a2b1-308eda0170f4\") " pod="openshift-dns/node-resolver-fmkwp"
Sep 11 17:20:30.927251 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.927189    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ccc7a17-e4f2-48ac-a2b1-308eda0170f4-tmp-dir\") pod \"node-resolver-fmkwp\" (UID: \"8ccc7a17-e4f2-48ac-a2b1-308eda0170f4\") " pod="openshift-dns/node-resolver-fmkwp"
Sep 11 17:20:30.927251 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.927240    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-system-cni-dir\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:30.927327 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.927266    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-cni-binary-copy\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:30.927327 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.927286    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:30.927327 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.927301    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6bn4\" (UniqueName: \"kubernetes.io/projected/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-kube-api-access-l6bn4\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:30.927327 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.927317    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltr7q\" (UniqueName: \"kubernetes.io/projected/8ccc7a17-e4f2-48ac-a2b1-308eda0170f4-kube-api-access-ltr7q\") pod \"node-resolver-fmkwp\" (UID: \"8ccc7a17-e4f2-48ac-a2b1-308eda0170f4\") " pod="openshift-dns/node-resolver-fmkwp"
Sep 11 17:20:30.927463 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.927382    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-cnibin\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:30.927463 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.927412    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:30.927463 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.927453    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:30.927621 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.927594    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:30.928595 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.928580    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 17:20:30.928658 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.928642    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ttqn4"
Sep 11 17:20:30.928790 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.928773    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vmr96\""
Sep 11 17:20:30.929068 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.929055    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 17:20:30.930162 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.930144    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:30.930229 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:30.930205    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:30.931307 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.931250    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b7t5c"
Sep 11 17:20:30.931406 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.931326    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 17:20:30.931406 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.931354    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 17:20:30.931520 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.931325    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:20:30.932106 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.932089    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 17:20:30.932239 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.932226    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ppmm6\""
Sep 11 17:20:30.932283 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.932271    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xg2nl\""
Sep 11 17:20:30.932587 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.932568    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:30.932697 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.932657    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 17:20:30.934020 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.934004    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:30.934105 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:30.934056    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:30.936066 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.935991    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:30.936206 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.936186    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 17:20:30.936336 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.936217    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:20:30.936412 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.936370    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:20:30.936832 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.936813    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 17:20:30.937295 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.937280    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 17:20:30.937429 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.937409    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 17:20:30.937429 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.937413    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 17:20:30.937667 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.937652    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 17:20:30.937747 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.937685    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 17:20:30.937791 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.937768    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 17:20:30.938069 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.938056    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vb66d\""
Sep 11 17:20:30.938117 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.938070    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z7hq6\""
Sep 11 17:20:30.940102 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.940079    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qwnzq\""
Sep 11 17:20:30.940441 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.940425    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 17:20:30.942809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.941617    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 17:20:30.942809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.941632    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 17:20:30.962524 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.962502    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fqc5p"
Sep 11 17:20:30.970362 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:30.970345    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fqc5p"
Sep 11 17:20:31.022493 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.022459    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 17:20:31.028305 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-sys\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.028409 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028317    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-system-cni-dir\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:31.028409 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6bn4\" (UniqueName: \"kubernetes.io/projected/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-kube-api-access-l6bn4\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:31.028409 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028353    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-ovnkube-script-lib\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.028409 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028374    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-kubernetes\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.028409 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028394    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-host\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.028409 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028400    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-system-cni-dir\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:31.028646 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028418    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e81d2ca8-8db5-4953-8006-560a9152e6f7-iptables-alerter-script\") pod \"iptables-alerter-b7t5c\" (UID: \"e81d2ca8-8db5-4953-8006-560a9152e6f7\") " pod="openshift-network-operator/iptables-alerter-b7t5c"
Sep 11 17:20:31.028646 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028468    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-etc-openvswitch\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.028646 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028523    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-tuned\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.028646 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028555    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bb63d6c-44b5-43ba-abfc-9642135096a9-tmp\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.028646 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028579    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ae8cc59-4697-44e8-9b49-3e97f11bdbb9-serviceca\") pod \"node-ca-ttqn4\" (UID: \"9ae8cc59-4697-44e8-9b49-3e97f11bdbb9\") " pod="openshift-image-registry/node-ca-ttqn4"
Sep 11 17:20:31.028646 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028615    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g8qg\" (UniqueName: \"kubernetes.io/projected/9ae8cc59-4697-44e8-9b49-3e97f11bdbb9-kube-api-access-7g8qg\") pod \"node-ca-ttqn4\" (UID: \"9ae8cc59-4697-44e8-9b49-3e97f11bdbb9\") " pod="openshift-image-registry/node-ca-ttqn4"
Sep 11 17:20:31.028876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028649    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ccc7a17-e4f2-48ac-a2b1-308eda0170f4-hosts-file\") pod \"node-resolver-fmkwp\" (UID: \"8ccc7a17-e4f2-48ac-a2b1-308eda0170f4\") " pod="openshift-dns/node-resolver-fmkwp"
Sep 11 17:20:31.028876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028667    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ccc7a17-e4f2-48ac-a2b1-308eda0170f4-tmp-dir\") pod \"node-resolver-fmkwp\" (UID: \"8ccc7a17-e4f2-48ac-a2b1-308eda0170f4\") " pod="openshift-dns/node-resolver-fmkwp"
Sep 11 17:20:31.028876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028681    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-cnibin\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.028876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-run-k8s-cni-cncf-io\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.028876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028727    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-var-lib-kubelet\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.028876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028753    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bc62979d-5f6d-45f6-b2a2-34cd2228a46e-agent-certs\") pod \"konnectivity-agent-5dl9t\" (UID: \"bc62979d-5f6d-45f6-b2a2-34cd2228a46e\") " pod="kube-system/konnectivity-agent-5dl9t"
Sep 11 17:20:31.028876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028757    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ccc7a17-e4f2-48ac-a2b1-308eda0170f4-hosts-file\") pod \"node-resolver-fmkwp\" (UID: \"8ccc7a17-e4f2-48ac-a2b1-308eda0170f4\") " pod="openshift-dns/node-resolver-fmkwp"
Sep 11 17:20:31.028876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028778    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.028876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028804    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-env-overrides\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.028876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028828    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:31.028876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028852    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-multus-conf-dir\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.028876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028876    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfvkx\" (UniqueName: \"kubernetes.io/projected/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-kube-api-access-mfvkx\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028901    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04be58fc-8937-4595-afcb-ab03d51fd1e7-kubelet-config\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028923    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028959    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-device-dir\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028981    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ccc7a17-e4f2-48ac-a2b1-308eda0170f4-tmp-dir\") pod \"node-resolver-fmkwp\" (UID: \"8ccc7a17-e4f2-48ac-a2b1-308eda0170f4\") " pod="openshift-dns/node-resolver-fmkwp"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.028989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-sys-fs\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029023    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kpnv\" (UniqueName: \"kubernetes.io/projected/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-kube-api-access-7kpnv\") pod \"network-metrics-daemon-8zghj\" (UID: \"d11d5fd7-89c6-4b5c-8340-9c75134f34bd\") " pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-kubelet\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029140    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-cni-netd\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-systemd\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029192    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-var-lib-kubelet\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029214    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e81d2ca8-8db5-4953-8006-560a9152e6f7-host-slash\") pod \"iptables-alerter-b7t5c\" (UID: \"e81d2ca8-8db5-4953-8006-560a9152e6f7\") " pod="openshift-network-operator/iptables-alerter-b7t5c"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029229    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029235    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04be58fc-8937-4595-afcb-ab03d51fd1e7-dbus\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029261    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-cnibin\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:31.029343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029287    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029306    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-cnibin\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-multus-socket-dir-parent\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-run-netns\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029359    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-run\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029446    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ae8cc59-4697-44e8-9b49-3e97f11bdbb9-host\") pod \"node-ca-ttqn4\" (UID: \"9ae8cc59-4697-44e8-9b49-3e97f11bdbb9\") " pod="openshift-image-registry/node-ca-ttqn4"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029493    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8d4m\" (UniqueName: \"kubernetes.io/projected/e81d2ca8-8db5-4953-8006-560a9152e6f7-kube-api-access-r8d4m\") pod \"iptables-alerter-b7t5c\" (UID: \"e81d2ca8-8db5-4953-8006-560a9152e6f7\") " pod="openshift-network-operator/iptables-alerter-b7t5c"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029519    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fcc70897-2929-4dba-a28a-0cb4eefc63f5-multus-daemon-config\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029542    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-run-systemd\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029564    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-log-socket\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029660    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-sysctl-conf\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029686    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fcc70897-2929-4dba-a28a-0cb4eefc63f5-cni-binary-copy\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029711    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029729    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-socket-dir\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029766    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-lib-modules\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.029872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029782    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-system-cni-dir\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029795    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-var-lib-cni-multus\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029809    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-run-netns\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvqlv\" (UniqueName: \"kubernetes.io/projected/3bb63d6c-44b5-43ba-abfc-9642135096a9-kube-api-access-mvqlv\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029860    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltr7q\" (UniqueName: \"kubernetes.io/projected/8ccc7a17-e4f2-48ac-a2b1-308eda0170f4-kube-api-access-ltr7q\") pod \"node-resolver-fmkwp\" (UID: \"8ccc7a17-e4f2-48ac-a2b1-308eda0170f4\") " pod="openshift-dns/node-resolver-fmkwp"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-var-lib-cni-bin\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.029978    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-slash\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030009    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-run-openvswitch\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030035    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-ovnkube-config\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030058    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030080    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-sysctl-d\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030096    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-node-log\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjtxl\" (UniqueName: \"kubernetes.io/projected/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-kube-api-access-mjtxl\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-systemd-units\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-cni-binary-copy\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030231    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-run-multus-certs\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.030546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030255    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-var-lib-openvswitch\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.031016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030279    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qctnt\" (UniqueName: \"kubernetes.io/projected/fcc70897-2929-4dba-a28a-0cb4eefc63f5-kube-api-access-qctnt\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.031016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030305    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-run-ovn\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.031016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030327    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-ovn-node-metrics-cert\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.031016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030351    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-registration-dir\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.031016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030376    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-modprobe-d\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.031016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-multus-cni-dir\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.031016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rsfh\" (UniqueName: \"kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh\") pod \"network-check-target-4lcjd\" (UID: \"848b2bb7-5d86-4f2a-8879-a48493e3983d\") " pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:31.031016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030447    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-cni-bin\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.031016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030469    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs\") pod \"network-metrics-daemon-8zghj\" (UID: \"d11d5fd7-89c6-4b5c-8340-9c75134f34bd\") " pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:31.031016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030512    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-hostroot\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.031016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030534    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-etc-kubernetes\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.031016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030556    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bc62979d-5f6d-45f6-b2a2-34cd2228a46e-konnectivity-ca\") pod \"konnectivity-agent-5dl9t\" (UID: \"bc62979d-5f6d-45f6-b2a2-34cd2228a46e\") " pod="kube-system/konnectivity-agent-5dl9t"
Sep 11 17:20:31.031016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030598    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.031016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030617    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-cni-binary-copy\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:31.031016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.030621    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-sysconfig\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.044907 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.044884    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 17:20:31.048271 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.048248    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6bn4\" (UniqueName: \"kubernetes.io/projected/e3359cff-2f12-4f6f-9e1a-4b4ea67af62b-kube-api-access-l6bn4\") pod \"multus-additional-cni-plugins-24d5m\" (UID: \"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b\") " pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:31.048271 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.048260    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltr7q\" (UniqueName: \"kubernetes.io/projected/8ccc7a17-e4f2-48ac-a2b1-308eda0170f4-kube-api-access-ltr7q\") pod \"node-resolver-fmkwp\" (UID: \"8ccc7a17-e4f2-48ac-a2b1-308eda0170f4\") " pod="openshift-dns/node-resolver-fmkwp"
Sep 11 17:20:31.095020 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:31.094995    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40235f875029f0595008c74b069417b4.slice/crio-41759c5c1c1c90c0a0efc199df21562253f40fa8a9c939044daa2417cf85689c WatchSource:0}: Error finding container 41759c5c1c1c90c0a0efc199df21562253f40fa8a9c939044daa2417cf85689c: Status 404 returned error can't find the container with id 41759c5c1c1c90c0a0efc199df21562253f40fa8a9c939044daa2417cf85689c
Sep 11 17:20:31.095394 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:31.095379    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcffbb3555202a1c58f0c3380c4a7532.slice/crio-a74635e68deef724a7e7557b774b16591e812742985cd3a514257e891c368c95 WatchSource:0}: Error finding container a74635e68deef724a7e7557b774b16591e812742985cd3a514257e891c368c95: Status 404 returned error can't find the container with id a74635e68deef724a7e7557b774b16591e812742985cd3a514257e891c368c95
Sep 11 17:20:31.101521 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.101507    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:20:31.131298 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131266    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-ovnkube-script-lib\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.131439 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131308    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-kubernetes\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.131439 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131330    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-host\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.131439 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131388    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-host\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.131439 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131419    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e81d2ca8-8db5-4953-8006-560a9152e6f7-iptables-alerter-script\") pod \"iptables-alerter-b7t5c\" (UID: \"e81d2ca8-8db5-4953-8006-560a9152e6f7\") " pod="openshift-network-operator/iptables-alerter-b7t5c"
Sep 11 17:20:31.131663 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131440    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-etc-openvswitch\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.131663 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131449    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-kubernetes\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.131663 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131461    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-tuned\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.131663 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131498    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bb63d6c-44b5-43ba-abfc-9642135096a9-tmp\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.131663 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ae8cc59-4697-44e8-9b49-3e97f11bdbb9-serviceca\") pod \"node-ca-ttqn4\" (UID: \"9ae8cc59-4697-44e8-9b49-3e97f11bdbb9\") " pod="openshift-image-registry/node-ca-ttqn4"
Sep 11 17:20:31.131663 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131542    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8qg\" (UniqueName: \"kubernetes.io/projected/9ae8cc59-4697-44e8-9b49-3e97f11bdbb9-kube-api-access-7g8qg\") pod \"node-ca-ttqn4\" (UID: \"9ae8cc59-4697-44e8-9b49-3e97f11bdbb9\") " pod="openshift-image-registry/node-ca-ttqn4"
Sep 11 17:20:31.131663 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131543    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-etc-openvswitch\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.131663 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-cnibin\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.131663 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-run-k8s-cni-cncf-io\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.131663 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131633    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-var-lib-kubelet\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.131663 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bc62979d-5f6d-45f6-b2a2-34cd2228a46e-agent-certs\") pod \"konnectivity-agent-5dl9t\" (UID: \"bc62979d-5f6d-45f6-b2a2-34cd2228a46e\") " pod="kube-system/konnectivity-agent-5dl9t"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131700    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-env-overrides\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131710    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-run-k8s-cni-cncf-io\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131724    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-multus-conf-dir\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfvkx\" (UniqueName: \"kubernetes.io/projected/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-kube-api-access-mfvkx\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131790    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-cnibin\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131791    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04be58fc-8937-4595-afcb-ab03d51fd1e7-kubelet-config\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131819    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131830    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131839    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04be58fc-8937-4595-afcb-ab03d51fd1e7-kubelet-config\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-device-dir\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131888    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-sys-fs\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131915    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kpnv\" (UniqueName: \"kubernetes.io/projected/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-kube-api-access-7kpnv\") pod \"network-metrics-daemon-8zghj\" (UID: \"d11d5fd7-89c6-4b5c-8340-9c75134f34bd\") " pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131940    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-kubelet\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131967    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-cni-netd\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.132248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131966    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-ovnkube-script-lib\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.131989    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-systemd\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132024    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ae8cc59-4697-44e8-9b49-3e97f11bdbb9-serviceca\") pod \"node-ca-ttqn4\" (UID: \"9ae8cc59-4697-44e8-9b49-3e97f11bdbb9\") " pod="openshift-image-registry/node-ca-ttqn4"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132053    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-var-lib-kubelet\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132099    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-multus-conf-dir\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132103    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-kubelet\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132133    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-cni-netd\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132143    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e81d2ca8-8db5-4953-8006-560a9152e6f7-host-slash\") pod \"iptables-alerter-b7t5c\" (UID: \"e81d2ca8-8db5-4953-8006-560a9152e6f7\") " pod="openshift-network-operator/iptables-alerter-b7t5c"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04be58fc-8937-4595-afcb-ab03d51fd1e7-dbus\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132181    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-systemd\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132197    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-multus-socket-dir-parent\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132218    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-device-dir\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132227    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-sys-fs\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132223    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-run-netns\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132220    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-env-overrides\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132270    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-var-lib-kubelet\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132275    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-var-lib-kubelet\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132286    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e81d2ca8-8db5-4953-8006-560a9152e6f7-host-slash\") pod \"iptables-alerter-b7t5c\" (UID: \"e81d2ca8-8db5-4953-8006-560a9152e6f7\") " pod="openshift-network-operator/iptables-alerter-b7t5c"
Sep 11 17:20:31.133127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132317    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-multus-socket-dir-parent\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132339    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-run\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132347    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04be58fc-8937-4595-afcb-ab03d51fd1e7-dbus\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132364    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-run-netns\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132369    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ae8cc59-4697-44e8-9b49-3e97f11bdbb9-host\") pod \"node-ca-ttqn4\" (UID: \"9ae8cc59-4697-44e8-9b49-3e97f11bdbb9\") " pod="openshift-image-registry/node-ca-ttqn4"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132392    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-run\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132396    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8d4m\" (UniqueName: \"kubernetes.io/projected/e81d2ca8-8db5-4953-8006-560a9152e6f7-kube-api-access-r8d4m\") pod \"iptables-alerter-b7t5c\" (UID: \"e81d2ca8-8db5-4953-8006-560a9152e6f7\") " pod="openshift-network-operator/iptables-alerter-b7t5c"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132422    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fcc70897-2929-4dba-a28a-0cb4eefc63f5-multus-daemon-config\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ae8cc59-4697-44e8-9b49-3e97f11bdbb9-host\") pod \"node-ca-ttqn4\" (UID: \"9ae8cc59-4697-44e8-9b49-3e97f11bdbb9\") " pod="openshift-image-registry/node-ca-ttqn4"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-run-systemd\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-log-socket\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132501    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-run-systemd\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132508    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-sysctl-conf\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fcc70897-2929-4dba-a28a-0cb4eefc63f5-cni-binary-copy\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132567    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132593    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-socket-dir\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132616    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-lib-modules\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132624    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-sysctl-conf\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.133954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132639    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-system-cni-dir\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132663    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-var-lib-cni-multus\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132672    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-log-socket\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132686    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-run-netns\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqlv\" (UniqueName: \"kubernetes.io/projected/3bb63d6c-44b5-43ba-abfc-9642135096a9-kube-api-access-mvqlv\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.132720    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132741    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-var-lib-cni-bin\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-slash\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132808    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-slash\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.132815    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret podName:04be58fc-8937-4595-afcb-ab03d51fd1e7 nodeName:}" failed. No retries permitted until 2026-09-11 17:20:31.632772908 +0000 UTC m=+2.065225688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret") pod "global-pull-secret-syncer-b8pb5" (UID: "04be58fc-8937-4595-afcb-ab03d51fd1e7") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.132917    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-socket-dir\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133006    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-lib-modules\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133003    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-run-openvswitch\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133059    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-system-cni-dir\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-ovnkube-config\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133100    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-var-lib-cni-multus\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133106    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e81d2ca8-8db5-4953-8006-560a9152e6f7-iptables-alerter-script\") pod \"iptables-alerter-b7t5c\" (UID: \"e81d2ca8-8db5-4953-8006-560a9152e6f7\") " pod="openshift-network-operator/iptables-alerter-b7t5c"
Sep 11 17:20:31.134655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133112    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-run-openvswitch\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133128    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133186    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fcc70897-2929-4dba-a28a-0cb4eefc63f5-multus-daemon-config\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133190    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133210    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fcc70897-2929-4dba-a28a-0cb4eefc63f5-cni-binary-copy\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133228    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-var-lib-cni-bin\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133259    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-run-netns\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-sysctl-d\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133316    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-node-log\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133339    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjtxl\" (UniqueName: \"kubernetes.io/projected/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-kube-api-access-mjtxl\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-systemd-units\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133390    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-run-multus-certs\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133399    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-sysctl-d\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-var-lib-openvswitch\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qctnt\" (UniqueName: \"kubernetes.io/projected/fcc70897-2929-4dba-a28a-0cb4eefc63f5-kube-api-access-qctnt\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133448    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-systemd-units\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133463    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-run-ovn\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.135145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133500    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-node-log\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-ovn-node-metrics-cert\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133533    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-registration-dir\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133539    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-var-lib-openvswitch\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133556    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-modprobe-d\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133576    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-host-run-multus-certs\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133577    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-ovnkube-config\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133614    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-run-ovn\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133660    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-multus-cni-dir\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133669    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-registration-dir\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133585    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-multus-cni-dir\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsfh\" (UniqueName: \"kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh\") pod \"network-check-target-4lcjd\" (UID: \"848b2bb7-5d86-4f2a-8879-a48493e3983d\") " pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133730    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-cni-bin\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133747    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-modprobe-d\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133773    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-cni-bin\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133845    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs\") pod \"network-metrics-daemon-8zghj\" (UID: \"d11d5fd7-89c6-4b5c-8340-9c75134f34bd\") " pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133870    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-hostroot\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.135659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133895    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-etc-kubernetes\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133918    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bc62979d-5f6d-45f6-b2a2-34cd2228a46e-konnectivity-ca\") pod \"konnectivity-agent-5dl9t\" (UID: \"bc62979d-5f6d-45f6-b2a2-34cd2228a46e\") " pod="kube-system/konnectivity-agent-5dl9t"
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.133946    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.134060    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs podName:d11d5fd7-89c6-4b5c-8340-9c75134f34bd nodeName:}" failed. No retries permitted until 2026-09-11 17:20:31.634043427 +0000 UTC m=+2.066496189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs") pod "network-metrics-daemon-8zghj" (UID: "d11d5fd7-89c6-4b5c-8340-9c75134f34bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133952    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-hostroot\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133975    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133945    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.134115    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-sysconfig\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.134138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-sys\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.133976    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcc70897-2929-4dba-a28a-0cb4eefc63f5-etc-kubernetes\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.134206    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-sys\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.134214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-sysconfig\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.134362    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bc62979d-5f6d-45f6-b2a2-34cd2228a46e-konnectivity-ca\") pod \"konnectivity-agent-5dl9t\" (UID: \"bc62979d-5f6d-45f6-b2a2-34cd2228a46e\") " pod="kube-system/konnectivity-agent-5dl9t"
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.134422    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bb63d6c-44b5-43ba-abfc-9642135096a9-tmp\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.134437    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3bb63d6c-44b5-43ba-abfc-9642135096a9-etc-tuned\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.135048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bc62979d-5f6d-45f6-b2a2-34cd2228a46e-agent-certs\") pod \"konnectivity-agent-5dl9t\" (UID: \"bc62979d-5f6d-45f6-b2a2-34cd2228a46e\") " pod="kube-system/konnectivity-agent-5dl9t"
Sep 11 17:20:31.136224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.135993    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-ovn-node-metrics-cert\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.146274 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.146242    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:20:31.146274 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.146269    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:20:31.146443 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.146283    2560 projected.go:196] Error preparing data for projected volume kube-api-access-6rsfh for pod openshift-network-diagnostics/network-check-target-4lcjd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:31.146443 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.146361    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh podName:848b2bb7-5d86-4f2a-8879-a48493e3983d nodeName:}" failed. No retries permitted until 2026-09-11 17:20:31.64634521 +0000 UTC m=+2.078797974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6rsfh" (UniqueName: "kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh") pod "network-check-target-4lcjd" (UID: "848b2bb7-5d86-4f2a-8879-a48493e3983d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:31.147437 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.147411    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8d4m\" (UniqueName: \"kubernetes.io/projected/e81d2ca8-8db5-4953-8006-560a9152e6f7-kube-api-access-r8d4m\") pod \"iptables-alerter-b7t5c\" (UID: \"e81d2ca8-8db5-4953-8006-560a9152e6f7\") " pod="openshift-network-operator/iptables-alerter-b7t5c"
Sep 11 17:20:31.147611 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.147519    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8qg\" (UniqueName: \"kubernetes.io/projected/9ae8cc59-4697-44e8-9b49-3e97f11bdbb9-kube-api-access-7g8qg\") pod \"node-ca-ttqn4\" (UID: \"9ae8cc59-4697-44e8-9b49-3e97f11bdbb9\") " pod="openshift-image-registry/node-ca-ttqn4"
Sep 11 17:20:31.147729 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.147711    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kpnv\" (UniqueName: \"kubernetes.io/projected/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-kube-api-access-7kpnv\") pod \"network-metrics-daemon-8zghj\" (UID: \"d11d5fd7-89c6-4b5c-8340-9c75134f34bd\") " pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:31.148592 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.148575    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfvkx\" (UniqueName: \"kubernetes.io/projected/1263d2ec-eaf2-44a9-b5db-64955c9b0f2f-kube-api-access-mfvkx\") pod \"ovnkube-node-mdw6t\" (UID: \"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.149629 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.149610    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:20:31.151195 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.151163    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjtxl\" (UniqueName: \"kubernetes.io/projected/6fc7ac5a-e840-4891-90c1-f2a392bae4f0-kube-api-access-mjtxl\") pod \"aws-ebs-csi-driver-node-s9xrh\" (UID: \"6fc7ac5a-e840-4891-90c1-f2a392bae4f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.151364 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.151349    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqlv\" (UniqueName: \"kubernetes.io/projected/3bb63d6c-44b5-43ba-abfc-9642135096a9-kube-api-access-mvqlv\") pod \"tuned-xlx9t\" (UID: \"3bb63d6c-44b5-43ba-abfc-9642135096a9\") " pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.151410 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.151350    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qctnt\" (UniqueName: \"kubernetes.io/projected/fcc70897-2929-4dba-a28a-0cb4eefc63f5-kube-api-access-qctnt\") pod \"multus-dv8fv\" (UID: \"fcc70897-2929-4dba-a28a-0cb4eefc63f5\") " pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.246647 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.246571    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fmkwp"
Sep 11 17:20:31.255646 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:31.255620    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ccc7a17_e4f2_48ac_a2b1_308eda0170f4.slice/crio-4535fc95b8e3ba4fbd43f6ceaa9a0120ce75ad262483af749ffe3658696c151a WatchSource:0}: Error finding container 4535fc95b8e3ba4fbd43f6ceaa9a0120ce75ad262483af749ffe3658696c151a: Status 404 returned error can't find the container with id 4535fc95b8e3ba4fbd43f6ceaa9a0120ce75ad262483af749ffe3658696c151a
Sep 11 17:20:31.268809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.268785    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-24d5m"
Sep 11 17:20:31.273301 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.273272    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dv8fv"
Sep 11 17:20:31.276715 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:31.276693    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3359cff_2f12_4f6f_9e1a_4b4ea67af62b.slice/crio-6061721e0b676cc94ab947b93706c827fd06b99f3335fdb4918fbf307ec044bd WatchSource:0}: Error finding container 6061721e0b676cc94ab947b93706c827fd06b99f3335fdb4918fbf307ec044bd: Status 404 returned error can't find the container with id 6061721e0b676cc94ab947b93706c827fd06b99f3335fdb4918fbf307ec044bd
Sep 11 17:20:31.276816 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.276760    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5dl9t"
Sep 11 17:20:31.282347 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:31.282322    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc70897_2929_4dba_a28a_0cb4eefc63f5.slice/crio-9fddc9b1a27db1ef35caeaf0821836f6db137daa67e57f3f99e53a8e69fba70f WatchSource:0}: Error finding container 9fddc9b1a27db1ef35caeaf0821836f6db137daa67e57f3f99e53a8e69fba70f: Status 404 returned error can't find the container with id 9fddc9b1a27db1ef35caeaf0821836f6db137daa67e57f3f99e53a8e69fba70f
Sep 11 17:20:31.287655 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:31.287636    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc62979d_5f6d_45f6_b2a2_34cd2228a46e.slice/crio-515e09798d21de0d7675e821893c3ece90051cbddd1860b51a8af166c45452b2 WatchSource:0}: Error finding container 515e09798d21de0d7675e821893c3ece90051cbddd1860b51a8af166c45452b2: Status 404 returned error can't find the container with id 515e09798d21de0d7675e821893c3ece90051cbddd1860b51a8af166c45452b2
Sep 11 17:20:31.305077 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.305061    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xlx9t"
Sep 11 17:20:31.313100 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:31.312941    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb63d6c_44b5_43ba_abfc_9642135096a9.slice/crio-a9bbb6cbc58d1bc08bd0521bbd15c7a25a67e989d1a75a46355e1b30528f53e1 WatchSource:0}: Error finding container a9bbb6cbc58d1bc08bd0521bbd15c7a25a67e989d1a75a46355e1b30528f53e1: Status 404 returned error can't find the container with id a9bbb6cbc58d1bc08bd0521bbd15c7a25a67e989d1a75a46355e1b30528f53e1
Sep 11 17:20:31.319792 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.319773    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ttqn4"
Sep 11 17:20:31.326963 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:31.326943    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae8cc59_4697_44e8_9b49_3e97f11bdbb9.slice/crio-0febc2a2af546f02e234803841ce48f834381f2cb0fde0d18ffeb34cace44a34 WatchSource:0}: Error finding container 0febc2a2af546f02e234803841ce48f834381f2cb0fde0d18ffeb34cace44a34: Status 404 returned error can't find the container with id 0febc2a2af546f02e234803841ce48f834381f2cb0fde0d18ffeb34cace44a34
Sep 11 17:20:31.334649 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.334634    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b7t5c"
Sep 11 17:20:31.342132 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:31.342117    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81d2ca8_8db5_4953_8006_560a9152e6f7.slice/crio-98cdb3d2129dcfb571ccdd3344b1f7184c6d48131d1483c4a211e4345f9e985f WatchSource:0}: Error finding container 98cdb3d2129dcfb571ccdd3344b1f7184c6d48131d1483c4a211e4345f9e985f: Status 404 returned error can't find the container with id 98cdb3d2129dcfb571ccdd3344b1f7184c6d48131d1483c4a211e4345f9e985f
Sep 11 17:20:31.353509 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.353494    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:31.357674 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.357656    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh"
Sep 11 17:20:31.363014 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:31.362995    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1263d2ec_eaf2_44a9_b5db_64955c9b0f2f.slice/crio-4f493b678d0e9dc3c574cc029e66df73816acd40ae9df6a9710799f8cc60c4a5 WatchSource:0}: Error finding container 4f493b678d0e9dc3c574cc029e66df73816acd40ae9df6a9710799f8cc60c4a5: Status 404 returned error can't find the container with id 4f493b678d0e9dc3c574cc029e66df73816acd40ae9df6a9710799f8cc60c4a5
Sep 11 17:20:31.365606 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:20:31.365585    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc7ac5a_e840_4891_90c1_f2a392bae4f0.slice/crio-f7e8fc0fd7fb20dd59471029699b2cd7c430f6af23385654b4eb19a767b81709 WatchSource:0}: Error finding container f7e8fc0fd7fb20dd59471029699b2cd7c430f6af23385654b4eb19a767b81709: Status 404 returned error can't find the container with id f7e8fc0fd7fb20dd59471029699b2cd7c430f6af23385654b4eb19a767b81709
Sep 11 17:20:31.638678 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.638607    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs\") pod \"network-metrics-daemon-8zghj\" (UID: \"d11d5fd7-89c6-4b5c-8340-9c75134f34bd\") " pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:31.638678 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.638652    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:31.638826 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.638732    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:31.638826 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.638734    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:31.638826 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.638781    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret podName:04be58fc-8937-4595-afcb-ab03d51fd1e7 nodeName:}" failed. No retries permitted until 2026-09-11 17:20:32.638767586 +0000 UTC m=+3.071220341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret") pod "global-pull-secret-syncer-b8pb5" (UID: "04be58fc-8937-4595-afcb-ab03d51fd1e7") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:31.638826 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.638795    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs podName:d11d5fd7-89c6-4b5c-8340-9c75134f34bd nodeName:}" failed. No retries permitted until 2026-09-11 17:20:32.638788375 +0000 UTC m=+3.071241129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs") pod "network-metrics-daemon-8zghj" (UID: "d11d5fd7-89c6-4b5c-8340-9c75134f34bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:31.720427 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.720394    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:20:31.739913 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.739883    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsfh\" (UniqueName: \"kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh\") pod \"network-check-target-4lcjd\" (UID: \"848b2bb7-5d86-4f2a-8879-a48493e3983d\") " pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:31.740099 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.740084    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:20:31.740169 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.740110    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:20:31.740169 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.740122    2560 projected.go:196] Error preparing data for projected volume kube-api-access-6rsfh for pod openshift-network-diagnostics/network-check-target-4lcjd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:31.740270 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:31.740174    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh podName:848b2bb7-5d86-4f2a-8879-a48493e3983d nodeName:}" failed. No retries permitted until 2026-09-11 17:20:32.740156728 +0000 UTC m=+3.172609490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6rsfh" (UniqueName: "kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh") pod "network-check-target-4lcjd" (UID: "848b2bb7-5d86-4f2a-8879-a48493e3983d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:31.972582 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.972448    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:15:30 +0000 UTC" deadline="2028-02-23 01:32:58.510207401 +0000 UTC"
Sep 11 17:20:31.972582 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:31.972522    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12704h12m26.537689669s"
Sep 11 17:20:32.044225 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.044200    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:20:32.047550 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.046820    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:32.047550 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:32.046927    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:32.047550 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.047266    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:32.047550 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:32.047353    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:32.069216 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.069121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-148.ec2.internal" event={"ID":"fcffbb3555202a1c58f0c3380c4a7532","Type":"ContainerStarted","Data":"a74635e68deef724a7e7557b774b16591e812742985cd3a514257e891c368c95"}
Sep 11 17:20:32.088557 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.088309    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b7t5c" event={"ID":"e81d2ca8-8db5-4953-8006-560a9152e6f7","Type":"ContainerStarted","Data":"98cdb3d2129dcfb571ccdd3344b1f7184c6d48131d1483c4a211e4345f9e985f"}
Sep 11 17:20:32.106620 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.106551    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ttqn4" event={"ID":"9ae8cc59-4697-44e8-9b49-3e97f11bdbb9","Type":"ContainerStarted","Data":"0febc2a2af546f02e234803841ce48f834381f2cb0fde0d18ffeb34cace44a34"}
Sep 11 17:20:32.130550 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.130362    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dv8fv" event={"ID":"fcc70897-2929-4dba-a28a-0cb4eefc63f5","Type":"ContainerStarted","Data":"9fddc9b1a27db1ef35caeaf0821836f6db137daa67e57f3f99e53a8e69fba70f"}
Sep 11 17:20:32.142924 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.142900    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24d5m" event={"ID":"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b","Type":"ContainerStarted","Data":"6061721e0b676cc94ab947b93706c827fd06b99f3335fdb4918fbf307ec044bd"}
Sep 11 17:20:32.160875 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.160847    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh" event={"ID":"6fc7ac5a-e840-4891-90c1-f2a392bae4f0","Type":"ContainerStarted","Data":"f7e8fc0fd7fb20dd59471029699b2cd7c430f6af23385654b4eb19a767b81709"}
Sep 11 17:20:32.177703 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.177678    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t" event={"ID":"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f","Type":"ContainerStarted","Data":"4f493b678d0e9dc3c574cc029e66df73816acd40ae9df6a9710799f8cc60c4a5"}
Sep 11 17:20:32.204774 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.204741    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xlx9t" event={"ID":"3bb63d6c-44b5-43ba-abfc-9642135096a9","Type":"ContainerStarted","Data":"a9bbb6cbc58d1bc08bd0521bbd15c7a25a67e989d1a75a46355e1b30528f53e1"}
Sep 11 17:20:32.225830 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.225762    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5dl9t" event={"ID":"bc62979d-5f6d-45f6-b2a2-34cd2228a46e","Type":"ContainerStarted","Data":"515e09798d21de0d7675e821893c3ece90051cbddd1860b51a8af166c45452b2"}
Sep 11 17:20:32.258521 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.258491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fmkwp" event={"ID":"8ccc7a17-e4f2-48ac-a2b1-308eda0170f4","Type":"ContainerStarted","Data":"4535fc95b8e3ba4fbd43f6ceaa9a0120ce75ad262483af749ffe3658696c151a"}
Sep 11 17:20:32.273247 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.269625    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal" event={"ID":"40235f875029f0595008c74b069417b4","Type":"ContainerStarted","Data":"41759c5c1c1c90c0a0efc199df21562253f40fa8a9c939044daa2417cf85689c"}
Sep 11 17:20:32.646454 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.646363    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:32.646454 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.646432    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs\") pod \"network-metrics-daemon-8zghj\" (UID: \"d11d5fd7-89c6-4b5c-8340-9c75134f34bd\") " pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:32.646683 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:32.646578    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:32.646683 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:32.646649    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs podName:d11d5fd7-89c6-4b5c-8340-9c75134f34bd nodeName:}" failed. No retries permitted until 2026-09-11 17:20:34.646613615 +0000 UTC m=+5.079066382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs") pod "network-metrics-daemon-8zghj" (UID: "d11d5fd7-89c6-4b5c-8340-9c75134f34bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:32.647079 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:32.647061    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:32.647171 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:32.647111    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret podName:04be58fc-8937-4595-afcb-ab03d51fd1e7 nodeName:}" failed. No retries permitted until 2026-09-11 17:20:34.647097438 +0000 UTC m=+5.079550198 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret") pod "global-pull-secret-syncer-b8pb5" (UID: "04be58fc-8937-4595-afcb-ab03d51fd1e7") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:32.747306 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.747269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsfh\" (UniqueName: \"kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh\") pod \"network-check-target-4lcjd\" (UID: \"848b2bb7-5d86-4f2a-8879-a48493e3983d\") " pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:32.747526 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:32.747508    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:20:32.747606 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:32.747535    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:20:32.747606 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:32.747548    2560 projected.go:196] Error preparing data for projected volume kube-api-access-6rsfh for pod openshift-network-diagnostics/network-check-target-4lcjd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:32.747606 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:32.747603    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh podName:848b2bb7-5d86-4f2a-8879-a48493e3983d nodeName:}" failed. No retries permitted until 2026-09-11 17:20:34.747585163 +0000 UTC m=+5.180037932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6rsfh" (UniqueName: "kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh") pod "network-check-target-4lcjd" (UID: "848b2bb7-5d86-4f2a-8879-a48493e3983d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:32.973047 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.972922    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:15:30 +0000 UTC" deadline="2028-02-06 21:00:22.593389928 +0000 UTC"
Sep 11 17:20:32.973047 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:32.972971    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12315h39m49.620423014s"
Sep 11 17:20:33.044909 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:33.044810    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:33.045102 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:33.044969    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:33.544821 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:33.544791    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:20:34.047216 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:34.047184    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:34.047686 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:34.047326    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:34.047791 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:34.047772    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:34.047921 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:34.047899    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:34.663448 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:34.663414    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs\") pod \"network-metrics-daemon-8zghj\" (UID: \"d11d5fd7-89c6-4b5c-8340-9c75134f34bd\") " pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:34.663627 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:34.663507    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:34.663701 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:34.663653    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:34.663777 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:34.663719    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret podName:04be58fc-8937-4595-afcb-ab03d51fd1e7 nodeName:}" failed. No retries permitted until 2026-09-11 17:20:38.663701751 +0000 UTC m=+9.096154509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret") pod "global-pull-secret-syncer-b8pb5" (UID: "04be58fc-8937-4595-afcb-ab03d51fd1e7") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:34.664139 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:34.664116    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:34.664236 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:34.664177    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs podName:d11d5fd7-89c6-4b5c-8340-9c75134f34bd nodeName:}" failed. No retries permitted until 2026-09-11 17:20:38.664161791 +0000 UTC m=+9.096614549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs") pod "network-metrics-daemon-8zghj" (UID: "d11d5fd7-89c6-4b5c-8340-9c75134f34bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:34.763904 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:34.763868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsfh\" (UniqueName: \"kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh\") pod \"network-check-target-4lcjd\" (UID: \"848b2bb7-5d86-4f2a-8879-a48493e3983d\") " pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:34.764073 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:34.764057    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:20:34.764138 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:34.764080    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:20:34.764138 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:34.764099    2560 projected.go:196] Error preparing data for projected volume kube-api-access-6rsfh for pod openshift-network-diagnostics/network-check-target-4lcjd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:34.764236 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:34.764163    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh podName:848b2bb7-5d86-4f2a-8879-a48493e3983d nodeName:}" failed. No retries permitted until 2026-09-11 17:20:38.764144281 +0000 UTC m=+9.196597048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6rsfh" (UniqueName: "kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh") pod "network-check-target-4lcjd" (UID: "848b2bb7-5d86-4f2a-8879-a48493e3983d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:35.045233 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:35.045194    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:35.045515 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:35.045485    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:36.046097 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:36.045587    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:36.046097 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:36.045720    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:36.046880 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:36.046729    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:36.046880 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:36.046829    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:37.045240 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:37.045205    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:37.045409 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:37.045347    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:38.044780 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:38.044744    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:38.045234 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:38.044874    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:38.045533 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:38.045503    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:38.045657 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:38.045638    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:38.694879 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:38.694836    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs\") pod \"network-metrics-daemon-8zghj\" (UID: \"d11d5fd7-89c6-4b5c-8340-9c75134f34bd\") " pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:38.695055 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:38.694911    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:38.695055 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:38.695038    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:38.695146 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:38.695101    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret podName:04be58fc-8937-4595-afcb-ab03d51fd1e7 nodeName:}" failed. No retries permitted until 2026-09-11 17:20:46.695083752 +0000 UTC m=+17.127536510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret") pod "global-pull-secret-syncer-b8pb5" (UID: "04be58fc-8937-4595-afcb-ab03d51fd1e7") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:38.695146 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:38.695039    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:38.695240 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:38.695178    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs podName:d11d5fd7-89c6-4b5c-8340-9c75134f34bd nodeName:}" failed. No retries permitted until 2026-09-11 17:20:46.695158491 +0000 UTC m=+17.127611267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs") pod "network-metrics-daemon-8zghj" (UID: "d11d5fd7-89c6-4b5c-8340-9c75134f34bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:38.795815 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:38.795702    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsfh\" (UniqueName: \"kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh\") pod \"network-check-target-4lcjd\" (UID: \"848b2bb7-5d86-4f2a-8879-a48493e3983d\") " pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:38.796006 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:38.795890    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:20:38.796006 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:38.795918    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:20:38.796006 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:38.795935    2560 projected.go:196] Error preparing data for projected volume kube-api-access-6rsfh for pod openshift-network-diagnostics/network-check-target-4lcjd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:38.796006 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:38.796000    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh podName:848b2bb7-5d86-4f2a-8879-a48493e3983d nodeName:}" failed. No retries permitted until 2026-09-11 17:20:46.795981156 +0000 UTC m=+17.228433926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6rsfh" (UniqueName: "kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh") pod "network-check-target-4lcjd" (UID: "848b2bb7-5d86-4f2a-8879-a48493e3983d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:39.044811 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:39.044722    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:39.045342 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:39.044875    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:40.044898 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:40.044855    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:40.045376 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:40.045019    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:40.049136 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:40.047804    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:40.049136 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:40.047941    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:41.045311 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:41.045279    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:41.045765 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:41.045405    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:42.044916 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:42.044873    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:42.044916 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:42.044913    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:42.045166 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:42.045025    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:42.045166 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:42.045140    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:43.045125 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:43.045084    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:43.045542 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:43.045224    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:44.045038 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:44.044977    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:44.045201 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:44.045100    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:44.045201 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:44.045183    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:44.045578 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:44.045286    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:45.044616 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:45.044575    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:45.044788 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:45.044694    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:46.044499 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:46.044396    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:46.044967 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:46.044532    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:46.044967 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:46.044593    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:46.044967 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:46.044699    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:46.754211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:46.754181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:46.754434 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:46.754237    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs\") pod \"network-metrics-daemon-8zghj\" (UID: \"d11d5fd7-89c6-4b5c-8340-9c75134f34bd\") " pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:46.754434 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:46.754336    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:46.754434 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:46.754337    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:46.754434 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:46.754396    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs podName:d11d5fd7-89c6-4b5c-8340-9c75134f34bd nodeName:}" failed. No retries permitted until 2026-09-11 17:21:02.754377335 +0000 UTC m=+33.186830092 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs") pod "network-metrics-daemon-8zghj" (UID: "d11d5fd7-89c6-4b5c-8340-9c75134f34bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:20:46.754434 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:46.754415    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret podName:04be58fc-8937-4595-afcb-ab03d51fd1e7 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:02.754404879 +0000 UTC m=+33.186857640 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret") pod "global-pull-secret-syncer-b8pb5" (UID: "04be58fc-8937-4595-afcb-ab03d51fd1e7") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:20:46.854881 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:46.854850    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsfh\" (UniqueName: \"kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh\") pod \"network-check-target-4lcjd\" (UID: \"848b2bb7-5d86-4f2a-8879-a48493e3983d\") " pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:46.855017 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:46.855000    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:20:46.855061 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:46.855020    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:20:46.855061 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:46.855029    2560 projected.go:196] Error preparing data for projected volume kube-api-access-6rsfh for pod openshift-network-diagnostics/network-check-target-4lcjd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:46.855142 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:46.855077    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh podName:848b2bb7-5d86-4f2a-8879-a48493e3983d nodeName:}" failed. No retries permitted until 2026-09-11 17:21:02.855060756 +0000 UTC m=+33.287513514 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6rsfh" (UniqueName: "kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh") pod "network-check-target-4lcjd" (UID: "848b2bb7-5d86-4f2a-8879-a48493e3983d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:20:47.045446 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:47.045369    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:47.045870 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:47.045528    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:48.045071 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:48.045031    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:48.045253 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:48.045031    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:48.045253 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:48.045163    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:48.045368 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:48.045252    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:49.045110 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:49.045070    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:49.045598 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:49.045231    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:50.045501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:50.045429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:50.046035 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:50.045534    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:50.046035 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:50.045638    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:50.046035 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:50.045758    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:50.316055 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:50.315820    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t" event={"ID":"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f","Type":"ContainerStarted","Data":"83e4753eab140d009e8a9a040c65783bbc503ff4010f99e58b0a50046956cec8"}
Sep 11 17:20:50.316154 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:50.316121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t" event={"ID":"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f","Type":"ContainerStarted","Data":"f341d1837d9273d82ebe5de14d3287cdf542d559c5affc3495f8eaa4710ade90"}
Sep 11 17:20:50.316226 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:50.316185    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t" event={"ID":"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f","Type":"ContainerStarted","Data":"3c7da7908437a9e19b371549ce85d56d13c68b981446e2243c4e06f674a6c28e"}
Sep 11 17:20:50.317671 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:50.317585    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xlx9t" event={"ID":"3bb63d6c-44b5-43ba-abfc-9642135096a9","Type":"ContainerStarted","Data":"1b699feb58ab22649c0cf18a61dfe2c1375a291afcc1541da5c64fd391d7c2d3"}
Sep 11 17:20:50.323129 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:50.323103    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-148.ec2.internal" event={"ID":"fcffbb3555202a1c58f0c3380c4a7532","Type":"ContainerStarted","Data":"8aefff50e928bfacb22dd04a9c9f856e73eb57f0e7f83948a77a04c5e12b63c2"}
Sep 11 17:20:50.325729 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:50.325708    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dv8fv" event={"ID":"fcc70897-2929-4dba-a28a-0cb4eefc63f5","Type":"ContainerStarted","Data":"eeee3b5c183d05803fd5df8bb2423ea5588ffbd9e45ec9d6b5af62e3a71acf53"}
Sep 11 17:20:50.336340 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:50.336273    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xlx9t" podStartSLOduration=1.860278723 podStartE2EDuration="20.336257698s" totalImagesPullingTime="18.475978975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:50.336221694 +0000 UTC m=+20.768674470" watchObservedRunningTime="2026-09-11 17:20:50.336257698 +0000 UTC m=+20.768710476"
Sep 11 17:20:50.370301 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:50.370250    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-148.ec2.internal" podStartSLOduration=20.370231538 podStartE2EDuration="20.370231538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:50.353031096 +0000 UTC m=+20.785483871" watchObservedRunningTime="2026-09-11 17:20:50.370231538 +0000 UTC m=+20.802684315"
Sep 11 17:20:50.370633 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:50.370591    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dv8fv" podStartSLOduration=1.483126619 podStartE2EDuration="20.370580006s" totalImagesPullingTime="18.887453387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:50.369884379 +0000 UTC m=+20.802337156" watchObservedRunningTime="2026-09-11 17:20:50.370580006 +0000 UTC m=+20.803032782"
Sep 11 17:20:51.045134 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.044957    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:51.045268 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:51.045214    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:51.328951 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.328903    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5dl9t" event={"ID":"bc62979d-5f6d-45f6-b2a2-34cd2228a46e","Type":"ContainerStarted","Data":"314e6d70a18907b851b83e1ebe65df87d36367cee6b2b0a662b2370d3292aa2c"}
Sep 11 17:20:51.330342 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.330310    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fmkwp" event={"ID":"8ccc7a17-e4f2-48ac-a2b1-308eda0170f4","Type":"ContainerStarted","Data":"14c9f02e1342d4e01d49d6163bd8cf43940a026c281ec57a0883de0b3af868c4"}
Sep 11 17:20:51.331673 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.331650    2560 generic.go:356] "Generic (PLEG): container finished" podID="40235f875029f0595008c74b069417b4" containerID="164a377163c79aac320661d09b21f0ff4f0195d12ff4afe0d5e034c1c5ef36ae" exitCode=0
Sep 11 17:20:51.331830 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.331747    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal" event={"ID":"40235f875029f0595008c74b069417b4","Type":"ContainerDied","Data":"164a377163c79aac320661d09b21f0ff4f0195d12ff4afe0d5e034c1c5ef36ae"}
Sep 11 17:20:51.333237 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.333213    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b7t5c" event={"ID":"e81d2ca8-8db5-4953-8006-560a9152e6f7","Type":"ContainerStarted","Data":"cd7ae86362c382e5d6d63a49231b9848a1c864ea68a82099d2001726cee30986"}
Sep 11 17:20:51.334634 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.334617    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ttqn4" event={"ID":"9ae8cc59-4697-44e8-9b49-3e97f11bdbb9","Type":"ContainerStarted","Data":"75aa2c1b92b3110f5c1dd9e61ecc8c8fe202443984ca1e1f84d28a697d0cb0cf"}
Sep 11 17:20:51.336037 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.336018    2560 generic.go:356] "Generic (PLEG): container finished" podID="e3359cff-2f12-4f6f-9e1a-4b4ea67af62b" containerID="a379291711a11a310ec92a88d5a1766ec5b16f8a1ca9d58caa4b9cd7382299ec" exitCode=0
Sep 11 17:20:51.336137 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.336080    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24d5m" event={"ID":"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b","Type":"ContainerDied","Data":"a379291711a11a310ec92a88d5a1766ec5b16f8a1ca9d58caa4b9cd7382299ec"}
Sep 11 17:20:51.337525 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.337503    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh" event={"ID":"6fc7ac5a-e840-4891-90c1-f2a392bae4f0","Type":"ContainerStarted","Data":"99c2a9d1567a95f7fd62f1280645112dacf4cbf7a2a831f0d09be8a30fedb91d"}
Sep 11 17:20:51.340231 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.340216    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:20:51.340568 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.340552    2560 generic.go:356] "Generic (PLEG): container finished" podID="1263d2ec-eaf2-44a9-b5db-64955c9b0f2f" containerID="f341d1837d9273d82ebe5de14d3287cdf542d559c5affc3495f8eaa4710ade90" exitCode=1
Sep 11 17:20:51.340645 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.340629    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t" event={"ID":"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f","Type":"ContainerDied","Data":"f341d1837d9273d82ebe5de14d3287cdf542d559c5affc3495f8eaa4710ade90"}
Sep 11 17:20:51.340695 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.340683    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t" event={"ID":"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f","Type":"ContainerStarted","Data":"88dab887b0944e5182909fdac215504beebac7949f4345b7545be694d2806162"}
Sep 11 17:20:51.340733 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.340717    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t" event={"ID":"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f","Type":"ContainerStarted","Data":"81c482964440b89fd79ac52f664f9212ee8feaca14a204fc2376ee387915e360"}
Sep 11 17:20:51.340762 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.340751    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t" event={"ID":"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f","Type":"ContainerStarted","Data":"19441d8771adc9f7593eee9d9c6dd3705002312fa14a1cdc13544345da222bc2"}
Sep 11 17:20:51.344316 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.344283    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5dl9t" podStartSLOduration=2.873686884 podStartE2EDuration="21.344270878s" totalImagesPullingTime="18.470583994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:51.343889283 +0000 UTC m=+21.776342073" watchObservedRunningTime="2026-09-11 17:20:51.344270878 +0000 UTC m=+21.776723652"
Sep 11 17:20:51.396304 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.396245    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ttqn4" podStartSLOduration=6.989996164 podStartE2EDuration="21.396225835s" totalImagesPullingTime="14.406229671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:51.380299135 +0000 UTC m=+21.812751911" watchObservedRunningTime="2026-09-11 17:20:51.396225835 +0000 UTC m=+21.828678615"
Sep 11 17:20:51.396437 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.396392    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b7t5c" podStartSLOduration=2.956679496 podStartE2EDuration="21.396382672s" totalImagesPullingTime="18.439703176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:51.395547349 +0000 UTC m=+21.828000126" watchObservedRunningTime="2026-09-11 17:20:51.396382672 +0000 UTC m=+21.828835450"
Sep 11 17:20:51.428306 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.427921    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fmkwp" podStartSLOduration=2.901828213 podStartE2EDuration="21.427902968s" totalImagesPullingTime="18.526074755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:51.42785247 +0000 UTC m=+21.860305249" watchObservedRunningTime="2026-09-11 17:20:51.427902968 +0000 UTC m=+21.860355747"
Sep 11 17:20:51.874251 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.874100    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 17:20:51.981235 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.981123    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T17:20:51.874247621Z","UUID":"92b10c08-6b17-430a-a449-0a0e3913d027","Handler":null,"Name":"","Endpoint":""}
Sep 11 17:20:51.982815 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.982785    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 17:20:51.982946 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:51.982825    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 17:20:52.044914 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:52.044839    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:52.045052 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:52.044970    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:52.045052 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:52.045022    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:52.045122 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:52.045103    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:52.347305 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:52.347225    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh" event={"ID":"6fc7ac5a-e840-4891-90c1-f2a392bae4f0","Type":"ContainerStarted","Data":"d2a0d40044a316509064475df4b14b090d04ae07381a75c97ae5ddb41d401b77"}
Sep 11 17:20:52.349253 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:52.349220    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal" event={"ID":"40235f875029f0595008c74b069417b4","Type":"ContainerStarted","Data":"a1e5fd8c1c92891a9ba14dc49e5349f7e1ad21388476e7c0f367b7df9be54374"}
Sep 11 17:20:52.363888 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:52.363844    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-148.ec2.internal" podStartSLOduration=22.363831882 podStartE2EDuration="22.363831882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:52.363431629 +0000 UTC m=+22.795884404" watchObservedRunningTime="2026-09-11 17:20:52.363831882 +0000 UTC m=+22.796284659"
Sep 11 17:20:53.045022 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:53.044977    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:53.045153 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:53.045108    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:53.354582 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:53.354493    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh" event={"ID":"6fc7ac5a-e840-4891-90c1-f2a392bae4f0","Type":"ContainerStarted","Data":"5b44bbcce45413e6d2526c40e5098d8587779334c29c963b661371b62ef343a1"}
Sep 11 17:20:53.358055 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:53.358024    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:20:53.358495 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:53.358450    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t" event={"ID":"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f","Type":"ContainerStarted","Data":"59bf77cc1dff0e7e2db6e0acc5666d7ed274d522bbb41bedeb2ef1f59b528d3d"}
Sep 11 17:20:53.372942 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:53.372891    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9xrh" podStartSLOduration=2.688841851 podStartE2EDuration="23.372874616s" totalImagesPullingTime="20.684032765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:53.372546733 +0000 UTC m=+23.804999511" watchObservedRunningTime="2026-09-11 17:20:53.372874616 +0000 UTC m=+23.805327395"
Sep 11 17:20:54.045502 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:54.045457    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:54.045723 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:54.045457    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:54.045723 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:54.045580    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:54.045723 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:54.045636    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:55.045225 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:55.045194    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:55.045664 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:55.045318    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:55.397011 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:55.396945    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5dl9t"
Sep 11 17:20:55.397645 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:55.397627    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5dl9t"
Sep 11 17:20:56.045619 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:56.045363    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:56.045996 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:56.045364    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:56.045996 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:56.045723    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:56.045996 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:56.045829    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:56.367738 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:56.367648    2560 generic.go:356] "Generic (PLEG): container finished" podID="e3359cff-2f12-4f6f-9e1a-4b4ea67af62b" containerID="20895395af05b57911508949c38481a27a53d3652e3f214feed7d00574bf325d" exitCode=0
Sep 11 17:20:56.367738 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:56.367721    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24d5m" event={"ID":"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b","Type":"ContainerDied","Data":"20895395af05b57911508949c38481a27a53d3652e3f214feed7d00574bf325d"}
Sep 11 17:20:56.371268 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:56.371252    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:20:56.371732 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:56.371706    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t" event={"ID":"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f","Type":"ContainerStarted","Data":"8eddde9fb12801257cbf6c3227ad0a7405518f52f2f609fa7db0202345050af7"}
Sep 11 17:20:56.371927 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:56.371912    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5dl9t"
Sep 11 17:20:56.372096 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:56.372079    2560 scope.go:117] "RemoveContainer" containerID="f341d1837d9273d82ebe5de14d3287cdf542d559c5affc3495f8eaa4710ade90"
Sep 11 17:20:56.372287 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:56.372271    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5dl9t"
Sep 11 17:20:57.045153 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:57.045114    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:57.045310 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:57.045241    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:57.382217 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:57.382152    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:20:57.382880 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:57.382536    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t" event={"ID":"1263d2ec-eaf2-44a9-b5db-64955c9b0f2f","Type":"ContainerStarted","Data":"5a18e0deb1cbc420dbcd34a50496bac86157905acead3b0bdcbc0157a8371b91"}
Sep 11 17:20:57.382940 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:57.382901    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:57.383002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:57.382985    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:57.383056 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:57.383038    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:57.397756 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:57.397716    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:57.398634 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:57.398604    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:20:57.414432 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:57.414390    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t" podStartSLOduration=8.957238698 podStartE2EDuration="27.414374247s" totalImagesPullingTime="18.457135549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:20:57.413246734 +0000 UTC m=+27.845699558" watchObservedRunningTime="2026-09-11 17:20:57.414374247 +0000 UTC m=+27.846827024"
Sep 11 17:20:57.715178 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:57.715097    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b8pb5"]
Sep 11 17:20:57.715319 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:57.715205    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:57.715362 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:57.715315    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:57.715627 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:57.715609    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4lcjd"]
Sep 11 17:20:57.715710 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:57.715699    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:57.715787 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:57.715771    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:57.752040 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:57.752011    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8zghj"]
Sep 11 17:20:57.752168 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:57.752120    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:57.752219 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:57.752202    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:58.386935 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:58.386903    2560 generic.go:356] "Generic (PLEG): container finished" podID="e3359cff-2f12-4f6f-9e1a-4b4ea67af62b" containerID="e3066ae545f312b757b799fa6f80ea65289f0006977d1a686f002c87f6d8a6e6" exitCode=0
Sep 11 17:20:58.387622 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:58.386998    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24d5m" event={"ID":"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b","Type":"ContainerDied","Data":"e3066ae545f312b757b799fa6f80ea65289f0006977d1a686f002c87f6d8a6e6"}
Sep 11 17:20:59.045664 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:59.045386    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:20:59.045796 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:59.045392    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:20:59.045796 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:59.045392    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:20:59.045893 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:59.045787    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:20:59.045893 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:59.045823    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:20:59.046025 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:20:59.045891    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:20:59.391061 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:59.390974    2560 generic.go:356] "Generic (PLEG): container finished" podID="e3359cff-2f12-4f6f-9e1a-4b4ea67af62b" containerID="49bbb6bda4cbc7fc510334824bc727d80e8c870b75dced2e73433c2784e38406" exitCode=0
Sep 11 17:20:59.391402 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:20:59.391065    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24d5m" event={"ID":"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b","Type":"ContainerDied","Data":"49bbb6bda4cbc7fc510334824bc727d80e8c870b75dced2e73433c2784e38406"}
Sep 11 17:21:01.045077 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:01.045037    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:21:01.045576 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:01.045051    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:21:01.045576 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:01.045167    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:21:01.045576 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:01.045052    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:21:01.045576 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:01.045250    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:21:01.045576 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:01.045328    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:21:02.768185 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:02.768146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs\") pod \"network-metrics-daemon-8zghj\" (UID: \"d11d5fd7-89c6-4b5c-8340-9c75134f34bd\") " pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:21:02.768833 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:02.768203    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:21:02.768833 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:02.768310    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:21:02.768833 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:02.768312    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:21:02.768833 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:02.768371    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret podName:04be58fc-8937-4595-afcb-ab03d51fd1e7 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:34.768353141 +0000 UTC m=+65.200805917 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret") pod "global-pull-secret-syncer-b8pb5" (UID: "04be58fc-8937-4595-afcb-ab03d51fd1e7") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:21:02.768833 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:02.768390    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs podName:d11d5fd7-89c6-4b5c-8340-9c75134f34bd nodeName:}" failed. No retries permitted until 2026-09-11 17:21:34.768382268 +0000 UTC m=+65.200835023 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs") pod "network-metrics-daemon-8zghj" (UID: "d11d5fd7-89c6-4b5c-8340-9c75134f34bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:21:02.869166 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:02.869124    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsfh\" (UniqueName: \"kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh\") pod \"network-check-target-4lcjd\" (UID: \"848b2bb7-5d86-4f2a-8879-a48493e3983d\") " pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:21:02.869347 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:02.869267    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:21:02.869347 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:02.869291    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:21:02.869347 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:02.869304    2560 projected.go:196] Error preparing data for projected volume kube-api-access-6rsfh for pod openshift-network-diagnostics/network-check-target-4lcjd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:21:02.869495 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:02.869368    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh podName:848b2bb7-5d86-4f2a-8879-a48493e3983d nodeName:}" failed. No retries permitted until 2026-09-11 17:21:34.869349704 +0000 UTC m=+65.301802494 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6rsfh" (UniqueName: "kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh") pod "network-check-target-4lcjd" (UID: "848b2bb7-5d86-4f2a-8879-a48493e3983d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:21:03.044821 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.044736    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:21:03.045001 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.044737    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:21:03.045001 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:03.044873    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8zghj" podUID="d11d5fd7-89c6-4b5c-8340-9c75134f34bd"
Sep 11 17:21:03.045001 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.044737    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:21:03.045001 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:03.044937    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4lcjd" podUID="848b2bb7-5d86-4f2a-8879-a48493e3983d"
Sep 11 17:21:03.045216 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:03.045004    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b8pb5" podUID="04be58fc-8937-4595-afcb-ab03d51fd1e7"
Sep 11 17:21:03.804564 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.804531    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-148.ec2.internal" event="NodeReady"
Sep 11 17:21:03.805022 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.804683    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 17:21:03.882662 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.882632    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x2zzk"]
Sep 11 17:21:03.897854 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.897824    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zkwcj"]
Sep 11 17:21:03.898008 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.897940    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:03.901983 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.900896    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 17:21:03.901983 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.901791    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n6xk4\""
Sep 11 17:21:03.904749 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.904724    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 17:21:03.921804 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.921776    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zkwcj"]
Sep 11 17:21:03.921804 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.921809    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x2zzk"]
Sep 11 17:21:03.921995 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.921907    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zkwcj"
Sep 11 17:21:03.924825 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.924802    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-k28ks\""
Sep 11 17:21:03.925447 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.925427    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 17:21:03.925524 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.925429    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 17:21:03.925754 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.925737    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 17:21:03.978049 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.977946    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:03.978049 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.977984    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63bddd62-4518-490e-863d-f2d592e79ec3-tmp-dir\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:03.978229 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.978098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63bddd62-4518-490e-863d-f2d592e79ec3-config-volume\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:03.978229 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:03.978124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x56f8\" (UniqueName: \"kubernetes.io/projected/63bddd62-4518-490e-863d-f2d592e79ec3-kube-api-access-x56f8\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:04.079399 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.079362    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2afe3f73-882c-41a4-b26b-a4f3cb0e4264-cert\") pod \"ingress-canary-zkwcj\" (UID: \"2afe3f73-882c-41a4-b26b-a4f3cb0e4264\") " pod="openshift-ingress-canary/ingress-canary-zkwcj"
Sep 11 17:21:04.079604 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.079433    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63bddd62-4518-490e-863d-f2d592e79ec3-config-volume\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:04.079604 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.079463    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x56f8\" (UniqueName: \"kubernetes.io/projected/63bddd62-4518-490e-863d-f2d592e79ec3-kube-api-access-x56f8\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:04.079604 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.079522    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:04.079604 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.079547    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63bddd62-4518-490e-863d-f2d592e79ec3-tmp-dir\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:04.079604 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.079598    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jtjz\" (UniqueName: \"kubernetes.io/projected/2afe3f73-882c-41a4-b26b-a4f3cb0e4264-kube-api-access-4jtjz\") pod \"ingress-canary-zkwcj\" (UID: \"2afe3f73-882c-41a4-b26b-a4f3cb0e4264\") " pod="openshift-ingress-canary/ingress-canary-zkwcj"
Sep 11 17:21:04.079796 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:04.079667    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:21:04.079796 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:04.079743    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls podName:63bddd62-4518-490e-863d-f2d592e79ec3 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:04.57972344 +0000 UTC m=+35.012176204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls") pod "dns-default-x2zzk" (UID: "63bddd62-4518-490e-863d-f2d592e79ec3") : secret "dns-default-metrics-tls" not found
Sep 11 17:21:04.079865 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.079853    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63bddd62-4518-490e-863d-f2d592e79ec3-tmp-dir\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:04.080131 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.080114    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63bddd62-4518-490e-863d-f2d592e79ec3-config-volume\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:04.092562 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.092539    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x56f8\" (UniqueName: \"kubernetes.io/projected/63bddd62-4518-490e-863d-f2d592e79ec3-kube-api-access-x56f8\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:04.180202 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.180166    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2afe3f73-882c-41a4-b26b-a4f3cb0e4264-cert\") pod \"ingress-canary-zkwcj\" (UID: \"2afe3f73-882c-41a4-b26b-a4f3cb0e4264\") " pod="openshift-ingress-canary/ingress-canary-zkwcj"
Sep 11 17:21:04.180374 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.180331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jtjz\" (UniqueName: \"kubernetes.io/projected/2afe3f73-882c-41a4-b26b-a4f3cb0e4264-kube-api-access-4jtjz\") pod \"ingress-canary-zkwcj\" (UID: \"2afe3f73-882c-41a4-b26b-a4f3cb0e4264\") " pod="openshift-ingress-canary/ingress-canary-zkwcj"
Sep 11 17:21:04.182829 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.182808    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2afe3f73-882c-41a4-b26b-a4f3cb0e4264-cert\") pod \"ingress-canary-zkwcj\" (UID: \"2afe3f73-882c-41a4-b26b-a4f3cb0e4264\") " pod="openshift-ingress-canary/ingress-canary-zkwcj"
Sep 11 17:21:04.193240 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.193212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jtjz\" (UniqueName: \"kubernetes.io/projected/2afe3f73-882c-41a4-b26b-a4f3cb0e4264-kube-api-access-4jtjz\") pod \"ingress-canary-zkwcj\" (UID: \"2afe3f73-882c-41a4-b26b-a4f3cb0e4264\") " pod="openshift-ingress-canary/ingress-canary-zkwcj"
Sep 11 17:21:04.231340 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.231264    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zkwcj"
Sep 11 17:21:04.283287 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.283250    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:04.400082 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.399905    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:04.406532 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.406507    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24d5m" event={"ID":"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b","Type":"ContainerStarted","Data":"a15c5db9d3e54a4ddd50dccdfd1d926cc11ec4c963813c4a046f9d77879ad19b"}
Sep 11 17:21:04.546409 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.546380    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zkwcj"]
Sep 11 17:21:04.551295 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:21:04.551263    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2afe3f73_882c_41a4_b26b_a4f3cb0e4264.slice/crio-0b286bd5f0d49326700daf9e230d80b31a6679c2311b2a66f3c46a0328fd40df WatchSource:0}: Error finding container 0b286bd5f0d49326700daf9e230d80b31a6679c2311b2a66f3c46a0328fd40df: Status 404 returned error can't find the container with id 0b286bd5f0d49326700daf9e230d80b31a6679c2311b2a66f3c46a0328fd40df
Sep 11 17:21:04.583205 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:04.583178    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:04.583320 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:04.583306    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:21:04.583373 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:04.583363    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls podName:63bddd62-4518-490e-863d-f2d592e79ec3 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:05.583350076 +0000 UTC m=+36.015802836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls") pod "dns-default-x2zzk" (UID: "63bddd62-4518-490e-863d-f2d592e79ec3") : secret "dns-default-metrics-tls" not found
Sep 11 17:21:05.044880 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:05.044842    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:21:05.045566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:05.044941    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:21:05.045566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:05.044952    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:21:05.047846 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:05.047827    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:21:05.048814 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:05.048793    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:21:05.048977 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:05.048960    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f9zbq\""
Sep 11 17:21:05.049023 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:05.048986    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:21:05.049070 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:05.049037    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47wvr\""
Sep 11 17:21:05.049070 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:05.049046    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:21:05.411491 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:05.411392    2560 generic.go:356] "Generic (PLEG): container finished" podID="e3359cff-2f12-4f6f-9e1a-4b4ea67af62b" containerID="a15c5db9d3e54a4ddd50dccdfd1d926cc11ec4c963813c4a046f9d77879ad19b" exitCode=0
Sep 11 17:21:05.411659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:05.411487    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24d5m" event={"ID":"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b","Type":"ContainerDied","Data":"a15c5db9d3e54a4ddd50dccdfd1d926cc11ec4c963813c4a046f9d77879ad19b"}
Sep 11 17:21:05.412573 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:05.412548    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zkwcj" event={"ID":"2afe3f73-882c-41a4-b26b-a4f3cb0e4264","Type":"ContainerStarted","Data":"0b286bd5f0d49326700daf9e230d80b31a6679c2311b2a66f3c46a0328fd40df"}
Sep 11 17:21:05.593006 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:05.592971    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:05.593182 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:05.593160    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:21:05.593252 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:05.593238    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls podName:63bddd62-4518-490e-863d-f2d592e79ec3 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:07.593217497 +0000 UTC m=+38.025670322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls") pod "dns-default-x2zzk" (UID: "63bddd62-4518-490e-863d-f2d592e79ec3") : secret "dns-default-metrics-tls" not found
Sep 11 17:21:06.419684 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:06.419635    2560 generic.go:356] "Generic (PLEG): container finished" podID="e3359cff-2f12-4f6f-9e1a-4b4ea67af62b" containerID="07eb4b3111979a2c703301143b7958a7c6136dcbdb219f3c028dce77ea0c4dd6" exitCode=0
Sep 11 17:21:06.420216 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:06.419698    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24d5m" event={"ID":"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b","Type":"ContainerDied","Data":"07eb4b3111979a2c703301143b7958a7c6136dcbdb219f3c028dce77ea0c4dd6"}
Sep 11 17:21:07.180932 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:07.180886    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:07.253539 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:07.253501    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:07.425504 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:07.425410    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-24d5m" event={"ID":"e3359cff-2f12-4f6f-9e1a-4b4ea67af62b","Type":"ContainerStarted","Data":"d6035209323e1e6f49ad9efcaa6ba8f11d0c6770821b03dba59f9d9437ed63b4"}
Sep 11 17:21:07.450381 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:07.450333    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-24d5m" podStartSLOduration=7.926143992 podStartE2EDuration="37.450318521s" totalImagesPullingTime="29.524174529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:07.448760343 +0000 UTC m=+37.881213212" watchObservedRunningTime="2026-09-11 17:21:07.450318521 +0000 UTC m=+37.882771298"
Sep 11 17:21:07.589018 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:07.588966    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:07.611016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:07.610985    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:07.611154 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:07.611134    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:21:07.611207 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:07.611198    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls podName:63bddd62-4518-490e-863d-f2d592e79ec3 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:11.611183425 +0000 UTC m=+42.043636193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls") pod "dns-default-x2zzk" (UID: "63bddd62-4518-490e-863d-f2d592e79ec3") : secret "dns-default-metrics-tls" not found
Sep 11 17:21:07.672276 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:07.672239    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:08.106518 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:08.106257    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:08.202153 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:08.202125    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:08.429264 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:08.429190    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zkwcj" event={"ID":"2afe3f73-882c-41a4-b26b-a4f3cb0e4264","Type":"ContainerStarted","Data":"842e078e07996ff52899e77564c678c46aee6e989c13af119b993511de743247"}
Sep 11 17:21:08.446227 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:08.446185    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zkwcj" podStartSLOduration=2.293698279 podStartE2EDuration="5.446172173s" totalImagesPullingTime="3.152473894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:08.445580603 +0000 UTC m=+38.878033380" watchObservedRunningTime="2026-09-11 17:21:08.446172173 +0000 UTC m=+38.878624951"
Sep 11 17:21:10.171297 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:10.171265    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:10.274125 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:10.274080    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:11.637445 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:11.637412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:11.637810 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:11.637593    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:21:11.637810 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:11.637650    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls podName:63bddd62-4518-490e-863d-f2d592e79ec3 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:19.637636423 +0000 UTC m=+50.070089192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls") pod "dns-default-x2zzk" (UID: "63bddd62-4518-490e-863d-f2d592e79ec3") : secret "dns-default-metrics-tls" not found
Sep 11 17:21:12.727828 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:12.727797    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:14.513159 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:14.513121    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:14.583331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:14.583299    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:19.688591 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:19.688551    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:19.689043 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:19.688680    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:21:19.689043 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:19.688743    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls podName:63bddd62-4518-490e-863d-f2d592e79ec3 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:35.688729298 +0000 UTC m=+66.121182052 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls") pod "dns-default-x2zzk" (UID: "63bddd62-4518-490e-863d-f2d592e79ec3") : secret "dns-default-metrics-tls" not found
Sep 11 17:21:25.144681 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:25.144650    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:25.223883 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:25.223850    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:27.691304 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.691263    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z9sdw"]
Sep 11 17:21:27.694347 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.694324    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z9sdw"
Sep 11 17:21:27.698336 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.698310    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:21:27.699650 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.699605    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 11 17:21:27.699770 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.699661    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8lmft\""
Sep 11 17:21:27.712027 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.712005    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z9sdw"]
Sep 11 17:21:27.769258 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.769226    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"]
Sep 11 17:21:27.771257 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.771240    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:27.774850 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.774782    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 11 17:21:27.774850 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.774793    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gqvq6\""
Sep 11 17:21:27.774850 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.774793    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 17:21:27.775140 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.774928    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 17:21:27.775140 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.775016    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 11 17:21:27.783167 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.783148    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"]
Sep 11 17:21:27.841293 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.841263    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4ntb\" (UniqueName: \"kubernetes.io/projected/fcb7feab-00d8-4799-9647-43725d1cc29f-kube-api-access-j4ntb\") pod \"volume-data-source-validator-86695b7cc7-z9sdw\" (UID: \"fcb7feab-00d8-4799-9647-43725d1cc29f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z9sdw"
Sep 11 17:21:27.877696 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.877667    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-24qlf"]
Sep 11 17:21:27.879290 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.879274    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8648dc8856-csvbr"]
Sep 11 17:21:27.879405 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.879388    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:27.880865 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.880848    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:27.881645 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.881628    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 11 17:21:27.881881 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.881867    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 11 17:21:27.882146 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.882119    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 17:21:27.882146 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.882137    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 17:21:27.882449 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.882433    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8vzhs\""
Sep 11 17:21:27.883548 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.883533    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 11 17:21:27.883682 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.883653    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 11 17:21:27.883682 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.883662    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 17:21:27.883682 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.883671    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 11 17:21:27.883834 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.883694    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 11 17:21:27.883834 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.883662    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lq9gc\""
Sep 11 17:21:27.883834 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.883781    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 17:21:27.888723 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.888704    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 11 17:21:27.892104 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.892085    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-24qlf"]
Sep 11 17:21:27.899207 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.899184    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8648dc8856-csvbr"]
Sep 11 17:21:27.942213 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.942126    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/85d1d971-8409-4d04-a9c0-578ab1e8102a-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-j4htb\" (UID: \"85d1d971-8409-4d04-a9c0-578ab1e8102a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:27.942213 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.942194    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmqn2\" (UniqueName: \"kubernetes.io/projected/85d1d971-8409-4d04-a9c0-578ab1e8102a-kube-api-access-gmqn2\") pod \"cluster-monitoring-operator-6cd954b897-j4htb\" (UID: \"85d1d971-8409-4d04-a9c0-578ab1e8102a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:27.942453 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.942252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-j4htb\" (UID: \"85d1d971-8409-4d04-a9c0-578ab1e8102a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:27.942453 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.942301    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4ntb\" (UniqueName: \"kubernetes.io/projected/fcb7feab-00d8-4799-9647-43725d1cc29f-kube-api-access-j4ntb\") pod \"volume-data-source-validator-86695b7cc7-z9sdw\" (UID: \"fcb7feab-00d8-4799-9647-43725d1cc29f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z9sdw"
Sep 11 17:21:27.954122 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:27.952877    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4ntb\" (UniqueName: \"kubernetes.io/projected/fcb7feab-00d8-4799-9647-43725d1cc29f-kube-api-access-j4ntb\") pod \"volume-data-source-validator-86695b7cc7-z9sdw\" (UID: \"fcb7feab-00d8-4799-9647-43725d1cc29f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z9sdw"
Sep 11 17:21:28.002750 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.002722    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z9sdw"
Sep 11 17:21:28.042809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.042777    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-default-certificate\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:28.042809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.042812    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:28.043014 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.042850    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e353cc34-4593-4fa9-bba7-6509ecef06cc-tmp\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.043014 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.042874    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e353cc34-4593-4fa9-bba7-6509ecef06cc-trusted-ca-bundle\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.043014 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.042897    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e353cc34-4593-4fa9-bba7-6509ecef06cc-serving-cert\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.043014 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.042918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-stats-auth\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:28.043014 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.042936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e353cc34-4593-4fa9-bba7-6509ecef06cc-snapshots\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.043014 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.043001    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/85d1d971-8409-4d04-a9c0-578ab1e8102a-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-j4htb\" (UID: \"85d1d971-8409-4d04-a9c0-578ab1e8102a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:28.043282 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.043033    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:28.043282 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.043053    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5wx8\" (UniqueName: \"kubernetes.io/projected/594faf95-252b-4fef-ab68-b4ace396cdd7-kube-api-access-r5wx8\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:28.043282 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.043076    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmqn2\" (UniqueName: \"kubernetes.io/projected/85d1d971-8409-4d04-a9c0-578ab1e8102a-kube-api-access-gmqn2\") pod \"cluster-monitoring-operator-6cd954b897-j4htb\" (UID: \"85d1d971-8409-4d04-a9c0-578ab1e8102a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:28.043282 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.043117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-j4htb\" (UID: \"85d1d971-8409-4d04-a9c0-578ab1e8102a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:28.043282 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.043134    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxnqg\" (UniqueName: \"kubernetes.io/projected/e353cc34-4593-4fa9-bba7-6509ecef06cc-kube-api-access-qxnqg\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.043282 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.043167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e353cc34-4593-4fa9-bba7-6509ecef06cc-service-ca-bundle\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.043282 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:28.043251    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 17:21:28.043681 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:28.043326    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls podName:85d1d971-8409-4d04-a9c0-578ab1e8102a nodeName:}" failed. No retries permitted until 2026-09-11 17:21:28.54330995 +0000 UTC m=+58.975762718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-j4htb" (UID: "85d1d971-8409-4d04-a9c0-578ab1e8102a") : secret "cluster-monitoring-operator-tls" not found
Sep 11 17:21:28.043940 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.043915    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/85d1d971-8409-4d04-a9c0-578ab1e8102a-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-j4htb\" (UID: \"85d1d971-8409-4d04-a9c0-578ab1e8102a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:28.051081 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.051056    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:28.053654 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.053627    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmqn2\" (UniqueName: \"kubernetes.io/projected/85d1d971-8409-4d04-a9c0-578ab1e8102a-kube-api-access-gmqn2\") pod \"cluster-monitoring-operator-6cd954b897-j4htb\" (UID: \"85d1d971-8409-4d04-a9c0-578ab1e8102a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:28.143534 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.143502    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:28.143534 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.143535    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5wx8\" (UniqueName: \"kubernetes.io/projected/594faf95-252b-4fef-ab68-b4ace396cdd7-kube-api-access-r5wx8\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:28.143720 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.143581    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxnqg\" (UniqueName: \"kubernetes.io/projected/e353cc34-4593-4fa9-bba7-6509ecef06cc-kube-api-access-qxnqg\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.143720 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:28.143688    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle podName:594faf95-252b-4fef-ab68-b4ace396cdd7 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:28.64366746 +0000 UTC m=+59.076120214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle") pod "router-default-8648dc8856-csvbr" (UID: "594faf95-252b-4fef-ab68-b4ace396cdd7") : configmap references non-existent config key: service-ca.crt
Sep 11 17:21:28.143720 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.143715    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e353cc34-4593-4fa9-bba7-6509ecef06cc-service-ca-bundle\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.143852 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.143772    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-default-certificate\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:28.143852 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.143790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:28.143852 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.143815    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e353cc34-4593-4fa9-bba7-6509ecef06cc-tmp\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.143852 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.143833    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e353cc34-4593-4fa9-bba7-6509ecef06cc-trusted-ca-bundle\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.144042 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:28.143886    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 17:21:28.144042 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.143890    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e353cc34-4593-4fa9-bba7-6509ecef06cc-serving-cert\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.144042 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:28.143952    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs podName:594faf95-252b-4fef-ab68-b4ace396cdd7 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:28.643930987 +0000 UTC m=+59.076383756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs") pod "router-default-8648dc8856-csvbr" (UID: "594faf95-252b-4fef-ab68-b4ace396cdd7") : secret "router-metrics-certs-default" not found
Sep 11 17:21:28.144042 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.143978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-stats-auth\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:28.144042 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.144003    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e353cc34-4593-4fa9-bba7-6509ecef06cc-snapshots\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.144304 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.144262    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e353cc34-4593-4fa9-bba7-6509ecef06cc-tmp\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.144469 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.144425    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e353cc34-4593-4fa9-bba7-6509ecef06cc-service-ca-bundle\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.144650 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.144611    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e353cc34-4593-4fa9-bba7-6509ecef06cc-snapshots\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.144827 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.144808    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e353cc34-4593-4fa9-bba7-6509ecef06cc-trusted-ca-bundle\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.146256 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.146235    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-default-certificate\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:28.146608 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.146585    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e353cc34-4593-4fa9-bba7-6509ecef06cc-serving-cert\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.146693 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.146636    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-stats-auth\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:28.150138 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.150116    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:28.152401 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.152384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5wx8\" (UniqueName: \"kubernetes.io/projected/594faf95-252b-4fef-ab68-b4ace396cdd7-kube-api-access-r5wx8\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:28.152849 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.152832    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxnqg\" (UniqueName: \"kubernetes.io/projected/e353cc34-4593-4fa9-bba7-6509ecef06cc-kube-api-access-qxnqg\") pod \"insights-operator-5844b856db-24qlf\" (UID: \"e353cc34-4593-4fa9-bba7-6509ecef06cc\") " pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.188923 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.188907    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-24qlf"
Sep 11 17:21:28.268609 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.268581    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z9sdw"]
Sep 11 17:21:28.271076 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:21:28.271053    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb7feab_00d8_4799_9647_43725d1cc29f.slice/crio-850bc0875869f4e926d647368b295cf9faf3c283c58e594b2559deb37a01e15c WatchSource:0}: Error finding container 850bc0875869f4e926d647368b295cf9faf3c283c58e594b2559deb37a01e15c: Status 404 returned error can't find the container with id 850bc0875869f4e926d647368b295cf9faf3c283c58e594b2559deb37a01e15c
Sep 11 17:21:28.459401 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.459377    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-24qlf"]
Sep 11 17:21:28.461615 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:21:28.461587    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode353cc34_4593_4fa9_bba7_6509ecef06cc.slice/crio-ce057c456e690ae16fcde01b00cff09ecd269ff4f2e4e8fb0dd3f47a0917bc68 WatchSource:0}: Error finding container ce057c456e690ae16fcde01b00cff09ecd269ff4f2e4e8fb0dd3f47a0917bc68: Status 404 returned error can't find the container with id ce057c456e690ae16fcde01b00cff09ecd269ff4f2e4e8fb0dd3f47a0917bc68
Sep 11 17:21:28.473478 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.473455    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-24qlf" event={"ID":"e353cc34-4593-4fa9-bba7-6509ecef06cc","Type":"ContainerStarted","Data":"ce057c456e690ae16fcde01b00cff09ecd269ff4f2e4e8fb0dd3f47a0917bc68"}
Sep 11 17:21:28.474535 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.474514    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z9sdw" event={"ID":"fcb7feab-00d8-4799-9647-43725d1cc29f","Type":"ContainerStarted","Data":"850bc0875869f4e926d647368b295cf9faf3c283c58e594b2559deb37a01e15c"}
Sep 11 17:21:28.547206 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.547171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-j4htb\" (UID: \"85d1d971-8409-4d04-a9c0-578ab1e8102a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:28.547338 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:28.547313    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 17:21:28.547389 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:28.547383    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls podName:85d1d971-8409-4d04-a9c0-578ab1e8102a nodeName:}" failed. No retries permitted until 2026-09-11 17:21:29.547368302 +0000 UTC m=+59.979821060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-j4htb" (UID: "85d1d971-8409-4d04-a9c0-578ab1e8102a") : secret "cluster-monitoring-operator-tls" not found
Sep 11 17:21:28.648452 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.648417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:28.648625 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:28.648524    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:28.648625 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:28.648603    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle podName:594faf95-252b-4fef-ab68-b4ace396cdd7 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:29.648581538 +0000 UTC m=+60.081034297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle") pod "router-default-8648dc8856-csvbr" (UID: "594faf95-252b-4fef-ab68-b4ace396cdd7") : configmap references non-existent config key: service-ca.crt
Sep 11 17:21:28.648625 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:28.648609    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 17:21:28.648759 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:28.648665    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs podName:594faf95-252b-4fef-ab68-b4ace396cdd7 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:29.648641392 +0000 UTC m=+60.081094148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs") pod "router-default-8648dc8856-csvbr" (UID: "594faf95-252b-4fef-ab68-b4ace396cdd7") : secret "router-metrics-certs-default" not found
Sep 11 17:21:29.402581 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:29.402549    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdw6t"
Sep 11 17:21:29.556613 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:29.556577    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-j4htb\" (UID: \"85d1d971-8409-4d04-a9c0-578ab1e8102a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:29.556787 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:29.556701    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 17:21:29.556787 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:29.556770    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls podName:85d1d971-8409-4d04-a9c0-578ab1e8102a nodeName:}" failed. No retries permitted until 2026-09-11 17:21:31.556751041 +0000 UTC m=+61.989203818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-j4htb" (UID: "85d1d971-8409-4d04-a9c0-578ab1e8102a") : secret "cluster-monitoring-operator-tls" not found
Sep 11 17:21:29.560984 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:29.560956    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:29.657468 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:29.657409    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:29.657578 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:29.657539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:29.657616 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:29.657575    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle podName:594faf95-252b-4fef-ab68-b4ace396cdd7 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:31.657558972 +0000 UTC m=+62.090011727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle") pod "router-default-8648dc8856-csvbr" (UID: "594faf95-252b-4fef-ab68-b4ace396cdd7") : configmap references non-existent config key: service-ca.crt
Sep 11 17:21:29.657672 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:29.657633    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 17:21:29.657721 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:29.657680    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs podName:594faf95-252b-4fef-ab68-b4ace396cdd7 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:31.657664266 +0000 UTC m=+62.090117042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs") pod "router-default-8648dc8856-csvbr" (UID: "594faf95-252b-4fef-ab68-b4ace396cdd7") : secret "router-metrics-certs-default" not found
Sep 11 17:21:29.660858 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:29.660827    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:30.481584 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.481549    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z9sdw" event={"ID":"fcb7feab-00d8-4799-9647-43725d1cc29f","Type":"ContainerStarted","Data":"43f14477ee841763ca79829872c9b8393f4d930fd32eb5dbafcb9a307f1ebe59"}
Sep 11 17:21:30.501548 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.501509    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-z9sdw" podStartSLOduration=2.171900268 podStartE2EDuration="3.501493004s" totalImagesPullingTime="1.329592736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:30.501399577 +0000 UTC m=+60.933852355" watchObservedRunningTime="2026-09-11 17:21:30.501493004 +0000 UTC m=+60.933945778"
Sep 11 17:21:30.619760 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.619730    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:30.692845 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.692809    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"]
Sep 11 17:21:30.694779 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.694762    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"
Sep 11 17:21:30.699127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.699108    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2hm5q\""
Sep 11 17:21:30.699287 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.699270    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 11 17:21:30.699668 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.699651    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 11 17:21:30.708453 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.708434    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"]
Sep 11 17:21:30.716513 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.716469    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:30.820908 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.820820    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:30.866335 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.866301    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7xhfb\" (UID: \"ea721b9a-6947-44eb-b898-b331c3ad7d64\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"
Sep 11 17:21:30.866465 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.866351    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ea721b9a-6947-44eb-b898-b331c3ad7d64-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-7xhfb\" (UID: \"ea721b9a-6947-44eb-b898-b331c3ad7d64\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"
Sep 11 17:21:30.908670 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.908629    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:30.966989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.966949    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7xhfb\" (UID: \"ea721b9a-6947-44eb-b898-b331c3ad7d64\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"
Sep 11 17:21:30.967160 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.967006    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ea721b9a-6947-44eb-b898-b331c3ad7d64-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-7xhfb\" (UID: \"ea721b9a-6947-44eb-b898-b331c3ad7d64\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"
Sep 11 17:21:30.967160 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:30.967105    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 17:21:30.967266 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:30.967166    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert podName:ea721b9a-6947-44eb-b898-b331c3ad7d64 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:31.46715023 +0000 UTC m=+61.899603000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7xhfb" (UID: "ea721b9a-6947-44eb-b898-b331c3ad7d64") : secret "networking-console-plugin-cert" not found
Sep 11 17:21:30.967759 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:30.967737    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ea721b9a-6947-44eb-b898-b331c3ad7d64-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-7xhfb\" (UID: \"ea721b9a-6947-44eb-b898-b331c3ad7d64\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"
Sep 11 17:21:31.470727 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.470692    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7xhfb\" (UID: \"ea721b9a-6947-44eb-b898-b331c3ad7d64\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"
Sep 11 17:21:31.470896 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:31.470852    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 17:21:31.470937 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:31.470920    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert podName:ea721b9a-6947-44eb-b898-b331c3ad7d64 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:32.470901902 +0000 UTC m=+62.903354677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7xhfb" (UID: "ea721b9a-6947-44eb-b898-b331c3ad7d64") : secret "networking-console-plugin-cert" not found
Sep 11 17:21:31.485490 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.485451    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-24qlf" event={"ID":"e353cc34-4593-4fa9-bba7-6509ecef06cc","Type":"ContainerStarted","Data":"578cb2e432f4f74de491d1f58a7768283b15bda5eaafa18296ff443359e04d20"}
Sep 11 17:21:31.504240 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.504183    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-24qlf" podStartSLOduration=2.486142116 podStartE2EDuration="4.504159279s" totalImagesPullingTime="2.018017163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:31.503552807 +0000 UTC m=+61.936005585" watchObservedRunningTime="2026-09-11 17:21:31.504159279 +0000 UTC m=+61.936612057"
Sep 11 17:21:31.571171 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.571134    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-j4htb\" (UID: \"85d1d971-8409-4d04-a9c0-578ab1e8102a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:31.571356 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:31.571299    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 17:21:31.571419 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:31.571380    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls podName:85d1d971-8409-4d04-a9c0-578ab1e8102a nodeName:}" failed. No retries permitted until 2026-09-11 17:21:35.571359315 +0000 UTC m=+66.003812080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-j4htb" (UID: "85d1d971-8409-4d04-a9c0-578ab1e8102a") : secret "cluster-monitoring-operator-tls" not found
Sep 11 17:21:31.672648 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.672601    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:31.672838 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:31.672717    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 17:21:31.672838 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.672745    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:31.672838 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:31.672788    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs podName:594faf95-252b-4fef-ab68-b4ace396cdd7 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:35.672767535 +0000 UTC m=+66.105220313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs") pod "router-default-8648dc8856-csvbr" (UID: "594faf95-252b-4fef-ab68-b4ace396cdd7") : secret "router-metrics-certs-default" not found
Sep 11 17:21:31.673019 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:31.672857    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle podName:594faf95-252b-4fef-ab68-b4ace396cdd7 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:35.672842732 +0000 UTC m=+66.105295490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle") pod "router-default-8648dc8856-csvbr" (UID: "594faf95-252b-4fef-ab68-b4ace396cdd7") : configmap references non-existent config key: service-ca.crt
Sep 11 17:21:31.749632 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.749557    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c5qhl"]
Sep 11 17:21:31.751481 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.751455    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:31.754641 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.754619    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 17:21:31.754770 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.754619    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 17:21:31.754770 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.754684    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-blz79\""
Sep 11 17:21:31.766594 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.766567    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c5qhl"]
Sep 11 17:21:31.771725 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.771691    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:31.837825 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.837794    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:31.875052 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.875026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:31.875129 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.875066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-data-volume\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:31.875129 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.875092    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-crio-socket\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:31.875129 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.875110    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxkwv\" (UniqueName: \"kubernetes.io/projected/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-api-access-hxkwv\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:31.875226 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.875182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:31.975970 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.975936    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:31.975970 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.975969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-data-volume\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:31.976152 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.975994    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-crio-socket\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:31.976152 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.976013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxkwv\" (UniqueName: \"kubernetes.io/projected/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-api-access-hxkwv\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:31.976152 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.976058    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:31.976152 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:31.976102    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:31.976152 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:31.976137    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:21:31.976305 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:31.976183    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls podName:ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:32.476162905 +0000 UTC m=+62.908615668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c5qhl" (UID: "ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:31.976305 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:31.976203    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm podName:ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:32.476194104 +0000 UTC m=+62.908646872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c5qhl" (UID: "ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9") : configmap "kube-rbac-proxy" not found
Sep 11 17:21:31.976305 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.976210    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-crio-socket\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:31.976305 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.976267    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-data-volume\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:31.984724 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:31.984696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxkwv\" (UniqueName: \"kubernetes.io/projected/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-api-access-hxkwv\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:32.479062 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:32.479013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:32.482349 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:32.479444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7xhfb\" (UID: \"ea721b9a-6947-44eb-b898-b331c3ad7d64\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"
Sep 11 17:21:32.482349 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:32.479543    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:21:32.482349 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:32.479612    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 17:21:32.482349 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:32.479651    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm podName:ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:33.479614274 +0000 UTC m=+63.912067049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c5qhl" (UID: "ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9") : configmap "kube-rbac-proxy" not found
Sep 11 17:21:32.482349 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:32.479681    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert podName:ea721b9a-6947-44eb-b898-b331c3ad7d64 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:34.479664526 +0000 UTC m=+64.912117296 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7xhfb" (UID: "ea721b9a-6947-44eb-b898-b331c3ad7d64") : secret "networking-console-plugin-cert" not found
Sep 11 17:21:32.482349 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:32.479803    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:32.482349 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:32.480023    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:32.482349 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:32.480070    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls podName:ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:33.480049548 +0000 UTC m=+63.912502317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c5qhl" (UID: "ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:33.023668 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:33.023630    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:33.138548 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:33.138511    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:33.209405 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:33.209369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:33.283658 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:33.283575    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:33.488753 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:33.488720    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:33.488933 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:33.488822    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:33.488933 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:33.488886    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:21:33.488933 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:33.488913    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:33.489044 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:33.488961    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm podName:ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:35.488940151 +0000 UTC m=+65.921392922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c5qhl" (UID: "ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9") : configmap "kube-rbac-proxy" not found
Sep 11 17:21:33.489044 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:33.488982    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls podName:ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:35.48897297 +0000 UTC m=+65.921425727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c5qhl" (UID: "ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:33.646703 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:33.646673    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:33.692434 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:33.692406    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fmkwp_8ccc7a17-e4f2-48ac-a2b1-308eda0170f4/dns-node-resolver/0.log"
Sep 11 17:21:33.749847 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:33.749814    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:33.941051 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:33.940970    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:34.041060 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:34.041022    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:34.495617 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:34.495581    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7xhfb\" (UID: \"ea721b9a-6947-44eb-b898-b331c3ad7d64\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"
Sep 11 17:21:34.495790 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:34.495722    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 17:21:34.495790 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:34.495781    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert podName:ea721b9a-6947-44eb-b898-b331c3ad7d64 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:38.495765938 +0000 UTC m=+68.928218708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7xhfb" (UID: "ea721b9a-6947-44eb-b898-b331c3ad7d64") : secret "networking-console-plugin-cert" not found
Sep 11 17:21:34.798284 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:34.798203    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:21:34.798284 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:34.798270    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs\") pod \"network-metrics-daemon-8zghj\" (UID: \"d11d5fd7-89c6-4b5c-8340-9c75134f34bd\") " pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:21:34.802001 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:34.801985    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:21:34.802085 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:34.801990    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:21:34.809454 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:34.809427    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 17:21:34.809561 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:34.809505    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs podName:d11d5fd7-89c6-4b5c-8340-9c75134f34bd nodeName:}" failed. No retries permitted until 2026-09-11 17:22:38.809483637 +0000 UTC m=+129.241936413 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs") pod "network-metrics-daemon-8zghj" (UID: "d11d5fd7-89c6-4b5c-8340-9c75134f34bd") : secret "metrics-daemon-secret" not found
Sep 11 17:21:34.811447 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:34.811425    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04be58fc-8937-4595-afcb-ab03d51fd1e7-original-pull-secret\") pod \"global-pull-secret-syncer-b8pb5\" (UID: \"04be58fc-8937-4595-afcb-ab03d51fd1e7\") " pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:21:34.891525 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:34.891501    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ttqn4_9ae8cc59-4697-44e8-9b49-3e97f11bdbb9/node-ca/0.log"
Sep 11 17:21:34.898980 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:34.898957    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsfh\" (UniqueName: \"kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh\") pod \"network-check-target-4lcjd\" (UID: \"848b2bb7-5d86-4f2a-8879-a48493e3983d\") " pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:21:34.902620 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:34.902603    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:21:34.912453 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:34.912434    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:21:34.923190 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:34.923159    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rsfh\" (UniqueName: \"kubernetes.io/projected/848b2bb7-5d86-4f2a-8879-a48493e3983d-kube-api-access-6rsfh\") pod \"network-check-target-4lcjd\" (UID: \"848b2bb7-5d86-4f2a-8879-a48493e3983d\") " pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:21:35.057935 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.057849    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47wvr\""
Sep 11 17:21:35.065530 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.065508    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:21:35.065633 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.065512    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b8pb5"
Sep 11 17:21:35.339464 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.339393    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b8pb5"]
Sep 11 17:21:35.342161 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:21:35.342136    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04be58fc_8937_4595_afcb_ab03d51fd1e7.slice/crio-d7ab45c25f846ad109d609556a02598ab085206b5ca3c60f04652fb85c202584 WatchSource:0}: Error finding container d7ab45c25f846ad109d609556a02598ab085206b5ca3c60f04652fb85c202584: Status 404 returned error can't find the container with id d7ab45c25f846ad109d609556a02598ab085206b5ca3c60f04652fb85c202584
Sep 11 17:21:35.347674 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.347651    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4lcjd"]
Sep 11 17:21:35.352284 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:21:35.352263    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848b2bb7_5d86_4f2a_8879_a48493e3983d.slice/crio-9b57a0bfbc843e8e568fc6aed76e73ad5304cc093de50fe24f27c949529f6afd WatchSource:0}: Error finding container 9b57a0bfbc843e8e568fc6aed76e73ad5304cc093de50fe24f27c949529f6afd: Status 404 returned error can't find the container with id 9b57a0bfbc843e8e568fc6aed76e73ad5304cc093de50fe24f27c949529f6afd
Sep 11 17:21:35.496382 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.496347    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4lcjd" event={"ID":"848b2bb7-5d86-4f2a-8879-a48493e3983d","Type":"ContainerStarted","Data":"9b57a0bfbc843e8e568fc6aed76e73ad5304cc093de50fe24f27c949529f6afd"}
Sep 11 17:21:35.497450 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.497426    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b8pb5" event={"ID":"04be58fc-8937-4595-afcb-ab03d51fd1e7","Type":"ContainerStarted","Data":"d7ab45c25f846ad109d609556a02598ab085206b5ca3c60f04652fb85c202584"}
Sep 11 17:21:35.503919 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.503900    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:35.503976 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.503964    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:35.504049 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:35.504034    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:35.504083 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:35.504058    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:21:35.504112 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:35.504091    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls podName:ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:39.504076542 +0000 UTC m=+69.936529296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c5qhl" (UID: "ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:35.504112 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:35.504104    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm podName:ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:39.504097333 +0000 UTC m=+69.936550088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c5qhl" (UID: "ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9") : configmap "kube-rbac-proxy" not found
Sep 11 17:21:35.593869 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.593795    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:35.604961 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.604938    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-j4htb\" (UID: \"85d1d971-8409-4d04-a9c0-578ab1e8102a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:35.605071 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:35.605058    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 17:21:35.605123 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:35.605114    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls podName:85d1d971-8409-4d04-a9c0-578ab1e8102a nodeName:}" failed. No retries permitted until 2026-09-11 17:21:43.60509906 +0000 UTC m=+74.037551816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-j4htb" (UID: "85d1d971-8409-4d04-a9c0-578ab1e8102a") : secret "cluster-monitoring-operator-tls" not found
Sep 11 17:21:35.690110 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.690081    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zkwcj_2afe3f73-882c-41a4-b26b-a4f3cb0e4264/serve-healthcheck-canary/0.log"
Sep 11 17:21:35.690267 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.690233    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:35.705600 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.705577    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:21:35.705645 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.705612    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:35.705688 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:35.705674    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:35.705746 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:35.705731    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:21:35.705797 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:35.705781    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 17:21:35.705833 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:35.705790    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls podName:63bddd62-4518-490e-863d-f2d592e79ec3 nodeName:}" failed. No retries permitted until 2026-09-11 17:22:07.705774165 +0000 UTC m=+98.138226943 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls") pod "dns-default-x2zzk" (UID: "63bddd62-4518-490e-863d-f2d592e79ec3") : secret "dns-default-metrics-tls" not found
Sep 11 17:21:35.705873 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:35.705840    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle podName:594faf95-252b-4fef-ab68-b4ace396cdd7 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:43.705828193 +0000 UTC m=+74.138280951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle") pod "router-default-8648dc8856-csvbr" (UID: "594faf95-252b-4fef-ab68-b4ace396cdd7") : configmap references non-existent config key: service-ca.crt
Sep 11 17:21:35.705873 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:35.705851    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs podName:594faf95-252b-4fef-ab68-b4ace396cdd7 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:43.705845096 +0000 UTC m=+74.138297850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs") pod "router-default-8648dc8856-csvbr" (UID: "594faf95-252b-4fef-ab68-b4ace396cdd7") : secret "router-metrics-certs-default" not found
Sep 11 17:21:36.725127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:36.725089    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:37.627097 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:37.627055    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:37.881605 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:37.881574    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:37.997981 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:37.997949    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:38.112276 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:38.112245    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:38.329180 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:38.329072    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:38.529073 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:38.529040    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7xhfb\" (UID: \"ea721b9a-6947-44eb-b898-b331c3ad7d64\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"
Sep 11 17:21:38.529250 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:38.529232    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 11 17:21:38.529332 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:38.529319    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert podName:ea721b9a-6947-44eb-b898-b331c3ad7d64 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:46.529295329 +0000 UTC m=+76.961748095 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7xhfb" (UID: "ea721b9a-6947-44eb-b898-b331c3ad7d64") : secret "networking-console-plugin-cert" not found
Sep 11 17:21:39.111678 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:39.111635    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:39.212670 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:39.212616    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:39.327052 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:39.326991    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:39.539199 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:39.539162    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:39.539398 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:39.539245    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:39.539398 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:39.539327    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:39.539398 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:39.539385    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:21:39.539550 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:39.539404    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls podName:ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:47.539382823 +0000 UTC m=+77.971835595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c5qhl" (UID: "ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:39.539550 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:39.539442    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm podName:ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9 nodeName:}" failed. No retries permitted until 2026-09-11 17:21:47.539428602 +0000 UTC m=+77.971881360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c5qhl" (UID: "ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9") : configmap "kube-rbac-proxy" not found
Sep 11 17:21:39.715266 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:39.715234    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:39.937674 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:39.937641    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:40.131747 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:40.131711    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:40.391158 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:40.391122    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:40.513436 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:40.513403    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4lcjd" event={"ID":"848b2bb7-5d86-4f2a-8879-a48493e3983d","Type":"ContainerStarted","Data":"15e432afda62757b34593b308b89865faa7bc0800815105d7e3418116b5c5206"}
Sep 11 17:21:40.513598 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:40.513567    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:21:40.515081 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:40.514991    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b8pb5" event={"ID":"04be58fc-8937-4595-afcb-ab03d51fd1e7","Type":"ContainerStarted","Data":"17991ad037e23c20170563d05d28ad1a977494c5ef15c6b7fc6bc4e9b6e83e3d"}
Sep 11 17:21:40.532847 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:40.532804    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4lcjd" podStartSLOduration=65.477481706 podStartE2EDuration="1m10.53279052s" totalImagesPullingTime="5.055308814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:40.532349989 +0000 UTC m=+70.964802766" watchObservedRunningTime="2026-09-11 17:21:40.53279052 +0000 UTC m=+70.965243296"
Sep 11 17:21:40.544220 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:40.544195    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:40.551875 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:40.551837    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b8pb5" podStartSLOduration=65.481858183 podStartE2EDuration="1m10.551823159s" totalImagesPullingTime="5.069964976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:40.550334295 +0000 UTC m=+70.982787072" watchObservedRunningTime="2026-09-11 17:21:40.551823159 +0000 UTC m=+70.984275936"
Sep 11 17:21:41.128841 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:41.128807    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:41.321172 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:41.321142    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:41.490610 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:41.490582    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:41.598859 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:41.598825    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:41.936036 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:41.936010    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:42.322369 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:42.322285    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:42.422540 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:42.422502    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:42.525259 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:42.525226    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:42.919668 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:42.919623    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:43.530243 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:43.530210    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:43.672167 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:43.672118    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-j4htb\" (UID: \"85d1d971-8409-4d04-a9c0-578ab1e8102a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:43.672337 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:43.672277    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 17:21:43.672378 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:43.672344    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls podName:85d1d971-8409-4d04-a9c0-578ab1e8102a nodeName:}" failed. No retries permitted until 2026-09-11 17:21:59.672326646 +0000 UTC m=+90.104779406 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-j4htb" (UID: "85d1d971-8409-4d04-a9c0-578ab1e8102a") : secret "cluster-monitoring-operator-tls" not found
Sep 11 17:21:43.773299 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:43.773243    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:43.773542 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:43.773331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:43.773947 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:43.773913    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/594faf95-252b-4fef-ab68-b4ace396cdd7-service-ca-bundle\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:43.775715 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:43.775694    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594faf95-252b-4fef-ab68-b4ace396cdd7-metrics-certs\") pod \"router-default-8648dc8856-csvbr\" (UID: \"594faf95-252b-4fef-ab68-b4ace396cdd7\") " pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:43.797375 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:43.797305    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lq9gc\""
Sep 11 17:21:43.804932 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:43.804911    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:43.839181 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:43.839147    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:43.935520 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:43.935457    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:44.072725 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:44.072626    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8648dc8856-csvbr"]
Sep 11 17:21:44.076877 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:21:44.076851    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594faf95_252b_4fef_ab68_b4ace396cdd7.slice/crio-4c19956821c964e92168fbcb4f93996b99f5ff03a416b4b68e9bd2b8fc189153 WatchSource:0}: Error finding container 4c19956821c964e92168fbcb4f93996b99f5ff03a416b4b68e9bd2b8fc189153: Status 404 returned error can't find the container with id 4c19956821c964e92168fbcb4f93996b99f5ff03a416b4b68e9bd2b8fc189153
Sep 11 17:21:44.338340 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:44.338252    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:44.529626 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:44.529592    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8648dc8856-csvbr" event={"ID":"594faf95-252b-4fef-ab68-b4ace396cdd7","Type":"ContainerStarted","Data":"725e639ef1219ad4ee86ddaf682df21578d4712e68aa6cab3d4fcbc06f708871"}
Sep 11 17:21:44.529851 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:44.529641    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8648dc8856-csvbr" event={"ID":"594faf95-252b-4fef-ab68-b4ace396cdd7","Type":"ContainerStarted","Data":"4c19956821c964e92168fbcb4f93996b99f5ff03a416b4b68e9bd2b8fc189153"}
Sep 11 17:21:44.548337 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:44.548293    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-8648dc8856-csvbr" podStartSLOduration=17.54827617 podStartE2EDuration="17.54827617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:44.547425887 +0000 UTC m=+74.979878663" watchObservedRunningTime="2026-09-11 17:21:44.54827617 +0000 UTC m=+74.980728946"
Sep 11 17:21:44.715066 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:44.715037    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:44.805487 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:44.805439    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:44.807990 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:44.807967    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:44.916531 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:44.916495    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:45.018520 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:45.018428    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:45.525187 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:45.525160    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:45.532437 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:45.532415    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:45.533588 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:45.533571    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8648dc8856-csvbr"
Sep 11 17:21:46.150048 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:46.150020    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:46.326513 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:46.326462    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:46.519896 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:46.519815    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:46.593422 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:46.593386    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7xhfb\" (UID: \"ea721b9a-6947-44eb-b898-b331c3ad7d64\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"
Sep 11 17:21:46.595653 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:46.595621    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea721b9a-6947-44eb-b898-b331c3ad7d64-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7xhfb\" (UID: \"ea721b9a-6947-44eb-b898-b331c3ad7d64\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"
Sep 11 17:21:46.603486 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:46.603456    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"
Sep 11 17:21:46.865679 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:46.865604    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb"]
Sep 11 17:21:46.869681 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:21:46.869655    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea721b9a_6947_44eb_b898_b331c3ad7d64.slice/crio-f1c7000aabc9d94caad6f862e2a9f7c498e9067d41ecd239c6992d46661c4974 WatchSource:0}: Error finding container f1c7000aabc9d94caad6f862e2a9f7c498e9067d41ecd239c6992d46661c4974: Status 404 returned error can't find the container with id f1c7000aabc9d94caad6f862e2a9f7c498e9067d41ecd239c6992d46661c4974
Sep 11 17:21:46.915279 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:46.915252    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:47.520342 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:47.520313    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:47.540260 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:47.540219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb" event={"ID":"ea721b9a-6947-44eb-b898-b331c3ad7d64","Type":"ContainerStarted","Data":"f1c7000aabc9d94caad6f862e2a9f7c498e9067d41ecd239c6992d46661c4974"}
Sep 11 17:21:47.602135 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:47.602102    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:47.602281 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:47.602214    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:47.602918 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:47.602559    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:47.602918 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:21:47.602646    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls podName:ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9 nodeName:}" failed. No retries permitted until 2026-09-11 17:22:03.602625789 +0000 UTC m=+94.035078559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c5qhl" (UID: "ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:21:47.602918 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:47.602654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:21:47.918870 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:47.918837    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:47.989910 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:47.989877    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:48.130761 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:48.130730    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:48.334917 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:48.334841    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:48.547063 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:48.547018    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb" event={"ID":"ea721b9a-6947-44eb-b898-b331c3ad7d64","Type":"ContainerStarted","Data":"83092adf7904feb487711f2195e42016bebe5e6667341c20b7a946bd6f4114c0"}
Sep 11 17:21:48.565084 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:48.565040    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7xhfb" podStartSLOduration=17.387638562 podStartE2EDuration="18.565024933s" totalImagesPullingTime="1.177386371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:48.564814615 +0000 UTC m=+78.997267393" watchObservedRunningTime="2026-09-11 17:21:48.565024933 +0000 UTC m=+78.997477734"
Sep 11 17:21:51.939809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:51.939768    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:52.038003 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:52.037973    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:52.104689 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:52.104577    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:52.234011 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:52.233936    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:52.417408 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:52.417379    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:52.536654 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:52.536584    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:52.610843 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:52.610818    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:52.808850 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:52.808785    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:52.869439 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:52.869411    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:53.820654 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:53.820621    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:54.018571 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:54.018545    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:54.387798 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:54.387770    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:54.634485 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:54.634436    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:54.698172 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:54.698100    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:58.370684 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.370648    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bc548f8d-pbfmb"]
Sep 11 17:21:58.374436 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.374411    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bc548f8d-pbfmb"
Sep 11 17:21:58.385353 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.385330    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 17:21:58.386498 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.386459    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 17:21:58.386498 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.386469    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 17:21:58.386649 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.386464    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 17:21:58.386649 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.386511    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5s7nl\""
Sep 11 17:21:58.397417 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.397383    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bc548f8d-pbfmb"]
Sep 11 17:21:58.471793 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.471757    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-rbnrs"]
Sep 11 17:21:58.474123 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.474106    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86565b78d5-ngh5h"]
Sep 11 17:21:58.474254 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.474238    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-rbnrs"
Sep 11 17:21:58.477829 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.476399    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmmp9\" (UniqueName: \"kubernetes.io/projected/a14a2cba-dc61-48ec-a462-4f79c6ff8858-kube-api-access-wmmp9\") pod \"managed-serviceaccount-addon-agent-57bc548f8d-pbfmb\" (UID: \"a14a2cba-dc61-48ec-a462-4f79c6ff8858\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bc548f8d-pbfmb"
Sep 11 17:21:58.477829 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.476448    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a14a2cba-dc61-48ec-a462-4f79c6ff8858-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57bc548f8d-pbfmb\" (UID: \"a14a2cba-dc61-48ec-a462-4f79c6ff8858\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bc548f8d-pbfmb"
Sep 11 17:21:58.478999 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.478166    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 17:21:58.478999 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.478230    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 17:21:58.478999 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.478167    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mz4mq\""
Sep 11 17:21:58.479934 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.479917    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.482390 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.482369    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 17:21:58.482503 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.482428    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 17:21:58.482691 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.482658    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nbpzt\""
Sep 11 17:21:58.482881 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.482861    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 17:21:58.490429 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.490408    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 17:21:58.495153 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.495133    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86565b78d5-ngh5h"]
Sep 11 17:21:58.496226 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.496207    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-rbnrs"]
Sep 11 17:21:58.511403 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.511382    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:58.577018 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.576995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f1440e1-fa51-49ec-9ec8-686d89bcc606-image-registry-private-configuration\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.577154 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.577036    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f1440e1-fa51-49ec-9ec8-686d89bcc606-bound-sa-token\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.577154 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.577058    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmmp9\" (UniqueName: \"kubernetes.io/projected/a14a2cba-dc61-48ec-a462-4f79c6ff8858-kube-api-access-wmmp9\") pod \"managed-serviceaccount-addon-agent-57bc548f8d-pbfmb\" (UID: \"a14a2cba-dc61-48ec-a462-4f79c6ff8858\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bc548f8d-pbfmb"
Sep 11 17:21:58.577154 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.577101    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a14a2cba-dc61-48ec-a462-4f79c6ff8858-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57bc548f8d-pbfmb\" (UID: \"a14a2cba-dc61-48ec-a462-4f79c6ff8858\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bc548f8d-pbfmb"
Sep 11 17:21:58.577264 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.577247    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f1440e1-fa51-49ec-9ec8-686d89bcc606-registry-certificates\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.577300 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.577276    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f1440e1-fa51-49ec-9ec8-686d89bcc606-ca-trust-extracted\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.577336 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.577324    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrbbp\" (UniqueName: \"kubernetes.io/projected/54ac5c72-1b5d-4813-a4c7-6f6f108752a2-kube-api-access-hrbbp\") pod \"downloads-76f99584f4-rbnrs\" (UID: \"54ac5c72-1b5d-4813-a4c7-6f6f108752a2\") " pod="openshift-console/downloads-76f99584f4-rbnrs"
Sep 11 17:21:58.577370 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.577345    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f1440e1-fa51-49ec-9ec8-686d89bcc606-trusted-ca\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.577370 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.577365    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f1440e1-fa51-49ec-9ec8-686d89bcc606-installation-pull-secrets\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.577431 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.577404    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f1440e1-fa51-49ec-9ec8-686d89bcc606-registry-tls\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.577462 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.577431    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d67cg\" (UniqueName: \"kubernetes.io/projected/5f1440e1-fa51-49ec-9ec8-686d89bcc606-kube-api-access-d67cg\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.579664 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.579636    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a14a2cba-dc61-48ec-a462-4f79c6ff8858-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57bc548f8d-pbfmb\" (UID: \"a14a2cba-dc61-48ec-a462-4f79c6ff8858\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bc548f8d-pbfmb"
Sep 11 17:21:58.594759 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.594731    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmmp9\" (UniqueName: \"kubernetes.io/projected/a14a2cba-dc61-48ec-a462-4f79c6ff8858-kube-api-access-wmmp9\") pod \"managed-serviceaccount-addon-agent-57bc548f8d-pbfmb\" (UID: \"a14a2cba-dc61-48ec-a462-4f79c6ff8858\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bc548f8d-pbfmb"
Sep 11 17:21:58.604520 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.604450    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:58.677846 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.677818    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrbbp\" (UniqueName: \"kubernetes.io/projected/54ac5c72-1b5d-4813-a4c7-6f6f108752a2-kube-api-access-hrbbp\") pod \"downloads-76f99584f4-rbnrs\" (UID: \"54ac5c72-1b5d-4813-a4c7-6f6f108752a2\") " pod="openshift-console/downloads-76f99584f4-rbnrs"
Sep 11 17:21:58.677980 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.677849    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f1440e1-fa51-49ec-9ec8-686d89bcc606-trusted-ca\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.677980 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.677866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f1440e1-fa51-49ec-9ec8-686d89bcc606-installation-pull-secrets\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.677980 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.677910    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f1440e1-fa51-49ec-9ec8-686d89bcc606-registry-tls\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.677980 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.677940    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d67cg\" (UniqueName: \"kubernetes.io/projected/5f1440e1-fa51-49ec-9ec8-686d89bcc606-kube-api-access-d67cg\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.678136 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.677976    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f1440e1-fa51-49ec-9ec8-686d89bcc606-image-registry-private-configuration\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.678136 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.678058    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f1440e1-fa51-49ec-9ec8-686d89bcc606-bound-sa-token\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.678229 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.678132    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f1440e1-fa51-49ec-9ec8-686d89bcc606-registry-certificates\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.678229 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.678168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f1440e1-fa51-49ec-9ec8-686d89bcc606-ca-trust-extracted\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.678568 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.678544    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f1440e1-fa51-49ec-9ec8-686d89bcc606-ca-trust-extracted\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.678888 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.678869    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f1440e1-fa51-49ec-9ec8-686d89bcc606-registry-certificates\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.678982 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.678962    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f1440e1-fa51-49ec-9ec8-686d89bcc606-trusted-ca\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.680453 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.680430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f1440e1-fa51-49ec-9ec8-686d89bcc606-image-registry-private-configuration\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.680772 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.680753    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f1440e1-fa51-49ec-9ec8-686d89bcc606-registry-tls\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.680826 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.680771    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f1440e1-fa51-49ec-9ec8-686d89bcc606-installation-pull-secrets\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.686391 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.686368    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrbbp\" (UniqueName: \"kubernetes.io/projected/54ac5c72-1b5d-4813-a4c7-6f6f108752a2-kube-api-access-hrbbp\") pod \"downloads-76f99584f4-rbnrs\" (UID: \"54ac5c72-1b5d-4813-a4c7-6f6f108752a2\") " pod="openshift-console/downloads-76f99584f4-rbnrs"
Sep 11 17:21:58.687690 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.687671    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f1440e1-fa51-49ec-9ec8-686d89bcc606-bound-sa-token\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.688041 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.688019    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d67cg\" (UniqueName: \"kubernetes.io/projected/5f1440e1-fa51-49ec-9ec8-686d89bcc606-kube-api-access-d67cg\") pod \"image-registry-86565b78d5-ngh5h\" (UID: \"5f1440e1-fa51-49ec-9ec8-686d89bcc606\") " pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.700866 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.700850    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bc548f8d-pbfmb"
Sep 11 17:21:58.704518 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.704497    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:58.786496 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.786442    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-rbnrs"
Sep 11 17:21:58.791181 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.791163    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:58.969851 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:58.969819    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bc548f8d-pbfmb"]
Sep 11 17:21:58.973209 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:21:58.973183    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda14a2cba_dc61_48ec_a462_4f79c6ff8858.slice/crio-e37345d48fbdb5b88cf63f99860d563c37ed453d52313773e371917e9b22d5bb WatchSource:0}: Error finding container e37345d48fbdb5b88cf63f99860d563c37ed453d52313773e371917e9b22d5bb: Status 404 returned error can't find the container with id e37345d48fbdb5b88cf63f99860d563c37ed453d52313773e371917e9b22d5bb
Sep 11 17:21:59.073148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:59.073119    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-rbnrs"]
Sep 11 17:21:59.076659 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:21:59.076638    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ac5c72_1b5d_4813_a4c7_6f6f108752a2.slice/crio-e2740f23eefc59cfdfef7fac8c2a0ecae80d404aac9b5d7fe2763602f9aee072 WatchSource:0}: Error finding container e2740f23eefc59cfdfef7fac8c2a0ecae80d404aac9b5d7fe2763602f9aee072: Status 404 returned error can't find the container with id e2740f23eefc59cfdfef7fac8c2a0ecae80d404aac9b5d7fe2763602f9aee072
Sep 11 17:21:59.093846 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:59.093826    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86565b78d5-ngh5h"]
Sep 11 17:21:59.095058 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:21:59.095039    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1440e1_fa51_49ec_9ec8_686d89bcc606.slice/crio-55ec9f5e5ad940f6a108a2c052ecc7860c1628219e669000ec39e6a2885ef862 WatchSource:0}: Error finding container 55ec9f5e5ad940f6a108a2c052ecc7860c1628219e669000ec39e6a2885ef862: Status 404 returned error can't find the container with id 55ec9f5e5ad940f6a108a2c052ecc7860c1628219e669000ec39e6a2885ef862
Sep 11 17:21:59.494945 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:59.494913    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:59.581543 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:59.581506    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86565b78d5-ngh5h" event={"ID":"5f1440e1-fa51-49ec-9ec8-686d89bcc606","Type":"ContainerStarted","Data":"9707840b9c7451830881c3c622e9a7fb50716c7fa54dc698ffa9ba1a4c13c8fd"}
Sep 11 17:21:59.581707 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:59.581566    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:21:59.581707 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:59.581588    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86565b78d5-ngh5h" event={"ID":"5f1440e1-fa51-49ec-9ec8-686d89bcc606","Type":"ContainerStarted","Data":"55ec9f5e5ad940f6a108a2c052ecc7860c1628219e669000ec39e6a2885ef862"}
Sep 11 17:21:59.582847 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:59.582820    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bc548f8d-pbfmb" event={"ID":"a14a2cba-dc61-48ec-a462-4f79c6ff8858","Type":"ContainerStarted","Data":"e37345d48fbdb5b88cf63f99860d563c37ed453d52313773e371917e9b22d5bb"}
Sep 11 17:21:59.584011 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:59.583992    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-rbnrs" event={"ID":"54ac5c72-1b5d-4813-a4c7-6f6f108752a2","Type":"ContainerStarted","Data":"e2740f23eefc59cfdfef7fac8c2a0ecae80d404aac9b5d7fe2763602f9aee072"}
Sep 11 17:21:59.593788 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:59.593764    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:21:59.604188 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:59.604152    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86565b78d5-ngh5h" podStartSLOduration=1.604139333 podStartE2EDuration="1.604139333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:21:59.602546573 +0000 UTC m=+90.034999357" watchObservedRunningTime="2026-09-11 17:21:59.604139333 +0000 UTC m=+90.036592109"
Sep 11 17:21:59.685063 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:59.685032    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-j4htb\" (UID: \"85d1d971-8409-4d04-a9c0-578ab1e8102a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:59.687646 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:59.687627    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/85d1d971-8409-4d04-a9c0-578ab1e8102a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-j4htb\" (UID: \"85d1d971-8409-4d04-a9c0-578ab1e8102a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:21:59.882280 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:59.882255    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gqvq6\""
Sep 11 17:21:59.890594 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:21:59.890563    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"
Sep 11 17:22:00.170014 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:00.169938    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb"]
Sep 11 17:22:00.173816 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:22:00.173781    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d1d971_8409_4d04_a9c0_578ab1e8102a.slice/crio-cdd3757f4896a599e0721a8ef42ed66fac16c9ff1e3877e386c9baac2ef0479f WatchSource:0}: Error finding container cdd3757f4896a599e0721a8ef42ed66fac16c9ff1e3877e386c9baac2ef0479f: Status 404 returned error can't find the container with id cdd3757f4896a599e0721a8ef42ed66fac16c9ff1e3877e386c9baac2ef0479f
Sep 11 17:22:00.588059 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:00.587968    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb" event={"ID":"85d1d971-8409-4d04-a9c0-578ab1e8102a","Type":"ContainerStarted","Data":"cdd3757f4896a599e0721a8ef42ed66fac16c9ff1e3877e386c9baac2ef0479f"}
Sep 11 17:22:02.049152 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.049127    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-647d54f9fd-s6g8m"]
Sep 11 17:22:02.052704 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.052552    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.055423 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.055395    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 17:22:02.056215 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.055699    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 17:22:02.056215 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.055921    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 17:22:02.056468 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.056449    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9tzbh\""
Sep 11 17:22:02.057146 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.056652    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 17:22:02.057146 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.056834    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 17:22:02.058117 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.058067    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-647d54f9fd-s6g8m"]
Sep 11 17:22:02.106095 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.106059    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4504947f-47bd-4fc2-b576-d43519a55ef6-console-serving-cert\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.106095 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.106104    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4504947f-47bd-4fc2-b576-d43519a55ef6-console-oauth-config\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.106334 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.106181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-console-config\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.106334 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.106220    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc2nv\" (UniqueName: \"kubernetes.io/projected/4504947f-47bd-4fc2-b576-d43519a55ef6-kube-api-access-wc2nv\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.106334 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.106257    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-oauth-serving-cert\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.106520 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.106407    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-service-ca\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.207253 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.207223    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-console-config\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.207367 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.207275    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc2nv\" (UniqueName: \"kubernetes.io/projected/4504947f-47bd-4fc2-b576-d43519a55ef6-kube-api-access-wc2nv\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.207689 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.207304    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-oauth-serving-cert\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.207689 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.207537    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-service-ca\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.207689 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.207610    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4504947f-47bd-4fc2-b576-d43519a55ef6-console-serving-cert\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.207689 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.207638    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4504947f-47bd-4fc2-b576-d43519a55ef6-console-oauth-config\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.207959 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.207940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-oauth-serving-cert\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.208061 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.208017    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-console-config\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.208361 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.208337    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-service-ca\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.210242 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.210218    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4504947f-47bd-4fc2-b576-d43519a55ef6-console-oauth-config\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.210640 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.210622    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4504947f-47bd-4fc2-b576-d43519a55ef6-console-serving-cert\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.215880 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.215863    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc2nv\" (UniqueName: \"kubernetes.io/projected/4504947f-47bd-4fc2-b576-d43519a55ef6-kube-api-access-wc2nv\") pod \"console-647d54f9fd-s6g8m\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") " pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.365002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.364925    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:02.553038 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.553001    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:02.596295 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.596264    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb" event={"ID":"85d1d971-8409-4d04-a9c0-578ab1e8102a","Type":"ContainerStarted","Data":"f5d37bcd30eeb607a57b90d4b0b00a79eaa7d877cef0a30e274815775c599425"}
Sep 11 17:22:02.612837 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.612762    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-j4htb" podStartSLOduration=33.619793778 podStartE2EDuration="35.612743167s" totalImagesPullingTime="1.992949389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:02.612097668 +0000 UTC m=+93.044550447" watchObservedRunningTime="2026-09-11 17:22:02.612743167 +0000 UTC m=+93.045195937"
Sep 11 17:22:02.623903 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.623837    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:02.649141 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.649116    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-647d54f9fd-s6g8m"]
Sep 11 17:22:02.651743 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:22:02.651710    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4504947f_47bd_4fc2_b576_d43519a55ef6.slice/crio-e0faa7d6512ecb001992999fdb6b790b98d4af7a979f2a533016f68ebb616ce0 WatchSource:0}: Error finding container e0faa7d6512ecb001992999fdb6b790b98d4af7a979f2a533016f68ebb616ce0: Status 404 returned error can't find the container with id e0faa7d6512ecb001992999fdb6b790b98d4af7a979f2a533016f68ebb616ce0
Sep 11 17:22:02.740298 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.740260    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:02.908589 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:02.908544    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:03.602405 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:03.602364    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647d54f9fd-s6g8m" event={"ID":"4504947f-47bd-4fc2-b576-d43519a55ef6","Type":"ContainerStarted","Data":"e0faa7d6512ecb001992999fdb6b790b98d4af7a979f2a533016f68ebb616ce0"}
Sep 11 17:22:03.621227 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:03.621194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:22:03.624126 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:03.624075    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c5qhl\" (UID: \"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9\") " pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:22:03.860665 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:03.860571    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c5qhl"
Sep 11 17:22:04.149767 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:04.149734    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c5qhl"]
Sep 11 17:22:04.153639 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:22:04.153607    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3cab1a_7e90_4ee0_b19c_c1541f17f0e9.slice/crio-f9a51496487714e90ae80cf27197656fcfd917509406de4abfc7aa09acf87a4f WatchSource:0}: Error finding container f9a51496487714e90ae80cf27197656fcfd917509406de4abfc7aa09acf87a4f: Status 404 returned error can't find the container with id f9a51496487714e90ae80cf27197656fcfd917509406de4abfc7aa09acf87a4f
Sep 11 17:22:04.608935 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:04.608848    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c5qhl" event={"ID":"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9","Type":"ContainerStarted","Data":"9e6176e637de298539d768377599bc37f601eee73ea750afe7534d31fed1f0ae"}
Sep 11 17:22:04.608935 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:04.608926    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c5qhl" event={"ID":"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9","Type":"ContainerStarted","Data":"f9a51496487714e90ae80cf27197656fcfd917509406de4abfc7aa09acf87a4f"}
Sep 11 17:22:05.780434 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:05.780398    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:05.847692 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:05.847654    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:06.311319 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:06.311286    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:06.618050 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:06.618011    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647d54f9fd-s6g8m" event={"ID":"4504947f-47bd-4fc2-b576-d43519a55ef6","Type":"ContainerStarted","Data":"930f68cc39d928a02a4df7172a6225762e24d72a9047e93e8479ff240ae67eac"}
Sep 11 17:22:06.620129 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:06.620102    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c5qhl" event={"ID":"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9","Type":"ContainerStarted","Data":"6f185a3067decf2b9d802bb40d9c5def02481b9dca9864001676d6390c66c4c3"}
Sep 11 17:22:06.643581 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:06.643525    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-647d54f9fd-s6g8m" podStartSLOduration=1.219626106 podStartE2EDuration="4.643507096s" totalImagesPullingTime="3.42388099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:06.642244508 +0000 UTC m=+97.074697286" watchObservedRunningTime="2026-09-11 17:22:06.643507096 +0000 UTC m=+97.075959871"
Sep 11 17:22:06.873833 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:06.873709    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:06.974682 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:06.974641    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:07.760852 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:07.760821    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:22:07.767465 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:07.767430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63bddd62-4518-490e-863d-f2d592e79ec3-metrics-tls\") pod \"dns-default-x2zzk\" (UID: \"63bddd62-4518-490e-863d-f2d592e79ec3\") " pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:22:07.814253 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:07.814224    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n6xk4\""
Sep 11 17:22:07.822358 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:07.822296    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:22:08.100205 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:08.100134    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x2zzk"]
Sep 11 17:22:08.103413 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:22:08.103384    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bddd62_4518_490e_863d_f2d592e79ec3.slice/crio-cad84c2b324812ad05247117f93ba8ed4092ac2c6317e3c992cdf59cf07d45c5 WatchSource:0}: Error finding container cad84c2b324812ad05247117f93ba8ed4092ac2c6317e3c992cdf59cf07d45c5: Status 404 returned error can't find the container with id cad84c2b324812ad05247117f93ba8ed4092ac2c6317e3c992cdf59cf07d45c5
Sep 11 17:22:08.629244 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:08.629206    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x2zzk" event={"ID":"63bddd62-4518-490e-863d-f2d592e79ec3","Type":"ContainerStarted","Data":"cad84c2b324812ad05247117f93ba8ed4092ac2c6317e3c992cdf59cf07d45c5"}
Sep 11 17:22:08.631697 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:08.631666    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c5qhl" event={"ID":"ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9","Type":"ContainerStarted","Data":"c3a7e9b8bedb510ec050d945780eced484b5a93b862371666e172f6f2b134197"}
Sep 11 17:22:08.656794 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:08.656745    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c5qhl" podStartSLOduration=34.390737553 podStartE2EDuration="37.65672787s" totalImagesPullingTime="3.265990317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:21:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:08.654960697 +0000 UTC m=+99.087413474" watchObservedRunningTime="2026-09-11 17:22:08.65672787 +0000 UTC m=+99.089180646"
Sep 11 17:22:08.732109 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:08.732071    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:10.647133 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:10.647098    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:10.762525 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:10.762487    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:10.969175 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:10.969091    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:11.198266 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.198231    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:11.262825 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.262744    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"]
Sep 11 17:22:11.266574 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.266541    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.269675 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.269597    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 17:22:11.270249 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.270219    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 17:22:11.274208 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.270794    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 17:22:11.274208 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.271099    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 17:22:11.274208 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.271340    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mmjlw\""
Sep 11 17:22:11.291097 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.291076    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"]
Sep 11 17:22:11.306110 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.306085    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rfbv6"]
Sep 11 17:22:11.309499 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.309464    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.311889 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.311712    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 17:22:11.311889 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.311795    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tszjt\""
Sep 11 17:22:11.313847 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.313820    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 17:22:11.313847 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.313842    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 17:22:11.319589 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.319566    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:11.394487 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.394447    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b337361-c434-4668-9ad2-a527b1789840-sys\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.394656 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.394492    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b337361-c434-4668-9ad2-a527b1789840-root\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.394656 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.394517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ece35c6-9894-448b-9e3c-c90566b9d95e-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.394656 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.394588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.394656 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.394609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ece35c6-9894-448b-9e3c-c90566b9d95e-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.394656 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.394627    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-accelerators-collector-config\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.394656 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.394652    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8485\" (UniqueName: \"kubernetes.io/projected/4b337361-c434-4668-9ad2-a527b1789840-kube-api-access-m8485\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.394903 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.394707    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-tls\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.394903 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.394769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-textfile\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.394903 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.394817    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99k82\" (UniqueName: \"kubernetes.io/projected/2ece35c6-9894-448b-9e3c-c90566b9d95e-kube-api-access-99k82\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.394903 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.394882    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b337361-c434-4668-9ad2-a527b1789840-metrics-client-ca\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.395062 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.394921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ece35c6-9894-448b-9e3c-c90566b9d95e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.395062 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.394942    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ece35c6-9894-448b-9e3c-c90566b9d95e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.395062 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.394959    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-wtmp\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.395062 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.394989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ece35c6-9894-448b-9e3c-c90566b9d95e-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.440618 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.440587    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:11.495698 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.495660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-textfile\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.495882 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.495729    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99k82\" (UniqueName: \"kubernetes.io/projected/2ece35c6-9894-448b-9e3c-c90566b9d95e-kube-api-access-99k82\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.495882 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.495780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b337361-c434-4668-9ad2-a527b1789840-metrics-client-ca\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.495882 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.495816    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ece35c6-9894-448b-9e3c-c90566b9d95e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.495882 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.495847    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ece35c6-9894-448b-9e3c-c90566b9d95e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.495882 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.495878    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-wtmp\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.496145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.495908    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ece35c6-9894-448b-9e3c-c90566b9d95e-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.496145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.495948    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b337361-c434-4668-9ad2-a527b1789840-sys\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.496145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.495970    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b337361-c434-4668-9ad2-a527b1789840-root\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.496145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.495997    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ece35c6-9894-448b-9e3c-c90566b9d95e-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.496145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.496031    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-textfile\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.496145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.496040    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.496145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.496104    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ece35c6-9894-448b-9e3c-c90566b9d95e-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.496145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.496130    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-accelerators-collector-config\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.496554 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.496157    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8485\" (UniqueName: \"kubernetes.io/projected/4b337361-c434-4668-9ad2-a527b1789840-kube-api-access-m8485\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.496554 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.496191    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-tls\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.496554 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:22:11.496293    2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 17:22:11.496554 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:22:11.496360    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-tls podName:4b337361-c434-4668-9ad2-a527b1789840 nodeName:}" failed. No retries permitted until 2026-09-11 17:22:11.996341583 +0000 UTC m=+102.428794338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-tls") pod "node-exporter-rfbv6" (UID: "4b337361-c434-4668-9ad2-a527b1789840") : secret "node-exporter-tls" not found
Sep 11 17:22:11.496554 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.496448    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-wtmp\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.496805 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.496634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b337361-c434-4668-9ad2-a527b1789840-sys\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.496805 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.496706    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b337361-c434-4668-9ad2-a527b1789840-root\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.497142 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.497074    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ece35c6-9894-448b-9e3c-c90566b9d95e-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.497266 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.497232    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-accelerators-collector-config\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.497332 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.497313    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ece35c6-9894-448b-9e3c-c90566b9d95e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.497394 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.497343    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ece35c6-9894-448b-9e3c-c90566b9d95e-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.497492 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.497452    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b337361-c434-4668-9ad2-a527b1789840-metrics-client-ca\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.499025 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.499000    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ece35c6-9894-448b-9e3c-c90566b9d95e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.499137 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.499031    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.499198 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.499140    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ece35c6-9894-448b-9e3c-c90566b9d95e-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.514335 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.514285    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99k82\" (UniqueName: \"kubernetes.io/projected/2ece35c6-9894-448b-9e3c-c90566b9d95e-kube-api-access-99k82\") pod \"kube-state-metrics-59fbdd77d-k4jzc\" (UID: \"2ece35c6-9894-448b-9e3c-c90566b9d95e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.515295 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.515279    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8485\" (UniqueName: \"kubernetes.io/projected/4b337361-c434-4668-9ad2-a527b1789840-kube-api-access-m8485\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:11.521133 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.521110    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4lcjd"
Sep 11 17:22:11.540585 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.540554    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:11.579048 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.579021    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"
Sep 11 17:22:11.640655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.640621    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:11.744050 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.744018    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:11.886364 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.886336    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:11.996706 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:11.996666    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:12.001872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.001833    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-tls\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:12.004248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.004229    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b337361-c434-4668-9ad2-a527b1789840-node-exporter-tls\") pod \"node-exporter-rfbv6\" (UID: \"4b337361-c434-4668-9ad2-a527b1789840\") " pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:12.219779 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.219685    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rfbv6"
Sep 11 17:22:12.236702 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.236656    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:12.365175 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.365139    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:12.365358 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.365203    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:12.366805 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.366771    2560 patch_prober.go:28] interesting pod/console-647d54f9fd-s6g8m container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused" start-of-body=
Sep 11 17:22:12.366933 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.366835    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-647d54f9fd-s6g8m" podUID="4504947f-47bd-4fc2-b576-d43519a55ef6" containerName="console" probeResult="failure" output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused"
Sep 11 17:22:12.398217 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.398185    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:22:12.412577 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.412550    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.425829 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.425806    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 17:22:12.427236 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.427213    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 17:22:12.427365 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.427346    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 17:22:12.427441 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.427395    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 17:22:12.427521 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.427447    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 17:22:12.427736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.427687    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 17:22:12.427844 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.427792    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 17:22:12.427844 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.427809    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xn6tb\""
Sep 11 17:22:12.428089 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.428072    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 17:22:12.428302 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.428207    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 17:22:12.459381 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.459350    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:12.467115 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.467089    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:22:12.505911 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.505794    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.505911 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.505883    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.506142 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.505913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-web-config\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.506142 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.505984    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.506142 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.506030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.506142 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.506094    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.506142 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.506126    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-config-out\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.506388 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.506148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.506388 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.506174    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.506388 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.506252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-config-volume\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.506388 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.506310    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fddzs\" (UniqueName: \"kubernetes.io/projected/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-kube-api-access-fddzs\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.506388 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.506353    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.506641 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.506388    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-tls-assets\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.607871 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.607825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.608055 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.607878    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-config-out\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.608055 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.607900    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.608055 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.607927    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.608055 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.607968    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-config-volume\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.608055 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.608003    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fddzs\" (UniqueName: \"kubernetes.io/projected/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-kube-api-access-fddzs\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.608055 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.608036    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.608706 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.608060    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-tls-assets\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.608706 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.608094    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.608706 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.608146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.608706 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.608169    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-web-config\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.608706 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.608225    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.608706 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.608249    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.608706 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:22:12.608487    2560 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 11 17:22:12.608706 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:22:12.608559    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-secret-alertmanager-main-tls podName:76883eb1-c32d-47d4-9832-5f59dc8fcdfe nodeName:}" failed. No retries permitted until 2026-09-11 17:22:13.10853931 +0000 UTC m=+103.540992079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "76883eb1-c32d-47d4-9832-5f59dc8fcdfe") : secret "alertmanager-main-tls" not found
Sep 11 17:22:12.608706 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:22:12.608591    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-alertmanager-trusted-ca-bundle podName:76883eb1-c32d-47d4-9832-5f59dc8fcdfe nodeName:}" failed. No retries permitted until 2026-09-11 17:22:13.108572962 +0000 UTC m=+103.541025734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "76883eb1-c32d-47d4-9832-5f59dc8fcdfe") : configmap references non-existent config key: ca-bundle.crt
Sep 11 17:22:12.609233 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.609115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.609937 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.609863    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.611735 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.611660    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-tls-assets\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.612346 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.612301    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-config-out\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.612424 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.612356    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-config-volume\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.612501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.612458    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.612754 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.612737    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.613505 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.613461    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.613597 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.613552    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-web-config\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.614236 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.614215    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.636178 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.636150    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fddzs\" (UniqueName: \"kubernetes.io/projected/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-kube-api-access-fddzs\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:12.796170 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:12.796090    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:13.113361 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:13.113280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:13.113361 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:13.113336    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:13.114099 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:13.114067    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:13.115971 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:13.115947    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/76883eb1-c32d-47d4-9832-5f59dc8fcdfe-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"76883eb1-c32d-47d4-9832-5f59dc8fcdfe\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:13.324253 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:13.324217    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:22:14.339137 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:14.339103    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:14.437919 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:14.437878    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:14.508082 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:14.508051    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:14.643997 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:14.643963    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:15.144226 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.144183    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:15.244971 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.244936    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:15.715009 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.714970    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-57c896d4fd-9hc4p"]
Sep 11 17:22:15.717851 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.717825    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.723344 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.723083    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mz2cz\""
Sep 11 17:22:15.723344 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.723107    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 17:22:15.723344 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.723146    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 17:22:15.723344 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.723168    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 17:22:15.723344 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.723147    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eoj8bl5tmjgg6\""
Sep 11 17:22:15.723344 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.723249    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 17:22:15.744182 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.744154    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57c896d4fd-9hc4p"]
Sep 11 17:22:15.842174 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.842136    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/76dde87e-bb30-4d65-9885-9fed9409ba76-secret-metrics-server-client-certs\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.842324 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.842212    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/76dde87e-bb30-4d65-9885-9fed9409ba76-audit-log\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.842324 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.842301    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76dde87e-bb30-4d65-9885-9fed9409ba76-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.842428 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.842353    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/76dde87e-bb30-4d65-9885-9fed9409ba76-secret-metrics-server-tls\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.842506 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.842425    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh6l2\" (UniqueName: \"kubernetes.io/projected/76dde87e-bb30-4d65-9885-9fed9409ba76-kube-api-access-qh6l2\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.842547 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.842520    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/76dde87e-bb30-4d65-9885-9fed9409ba76-metrics-server-audit-profiles\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.842587 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.842560    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/76dde87e-bb30-4d65-9885-9fed9409ba76-client-ca-bundle\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.944012 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.943974    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/76dde87e-bb30-4d65-9885-9fed9409ba76-secret-metrics-server-client-certs\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.944188 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.944044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/76dde87e-bb30-4d65-9885-9fed9409ba76-audit-log\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.944188 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.944097    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76dde87e-bb30-4d65-9885-9fed9409ba76-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.944188 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.944138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/76dde87e-bb30-4d65-9885-9fed9409ba76-secret-metrics-server-tls\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.944188 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.944162    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh6l2\" (UniqueName: \"kubernetes.io/projected/76dde87e-bb30-4d65-9885-9fed9409ba76-kube-api-access-qh6l2\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.944397 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.944205    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/76dde87e-bb30-4d65-9885-9fed9409ba76-metrics-server-audit-profiles\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.944397 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.944231    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/76dde87e-bb30-4d65-9885-9fed9409ba76-client-ca-bundle\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.945199 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.944607    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/76dde87e-bb30-4d65-9885-9fed9409ba76-audit-log\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.945916 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.945868    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/76dde87e-bb30-4d65-9885-9fed9409ba76-metrics-server-audit-profiles\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.946205 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.946166    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76dde87e-bb30-4d65-9885-9fed9409ba76-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.947664 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.947639    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/76dde87e-bb30-4d65-9885-9fed9409ba76-client-ca-bundle\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.947927 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.947902    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/76dde87e-bb30-4d65-9885-9fed9409ba76-secret-metrics-server-client-certs\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.948143 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.948121    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/76dde87e-bb30-4d65-9885-9fed9409ba76-secret-metrics-server-tls\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:15.960820 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:15.960761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh6l2\" (UniqueName: \"kubernetes.io/projected/76dde87e-bb30-4d65-9885-9fed9409ba76-kube-api-access-qh6l2\") pod \"metrics-server-57c896d4fd-9hc4p\" (UID: \"76dde87e-bb30-4d65-9885-9fed9409ba76\") " pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:16.030735 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.030646    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:16.034562 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.034526    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt"]
Sep 11 17:22:16.036989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.036965    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt"
Sep 11 17:22:16.039950 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.039923    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 17:22:16.040083 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.039931    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lc9mx\""
Sep 11 17:22:16.053102 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.053070    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt"]
Sep 11 17:22:16.056217 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.056160    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:16.146202 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.146159    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8336388-5be6-4a25-8a3c-110e377e84da-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5jfwt\" (UID: \"c8336388-5be6-4a25-8a3c-110e377e84da\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt"
Sep 11 17:22:16.146390 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.146290    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lb4b\" (UniqueName: \"kubernetes.io/projected/c8336388-5be6-4a25-8a3c-110e377e84da-kube-api-access-9lb4b\") pod \"monitoring-plugin-5d85778b9d-5jfwt\" (UID: \"c8336388-5be6-4a25-8a3c-110e377e84da\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt"
Sep 11 17:22:16.247600 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.247568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lb4b\" (UniqueName: \"kubernetes.io/projected/c8336388-5be6-4a25-8a3c-110e377e84da-kube-api-access-9lb4b\") pod \"monitoring-plugin-5d85778b9d-5jfwt\" (UID: \"c8336388-5be6-4a25-8a3c-110e377e84da\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt"
Sep 11 17:22:16.247765 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.247626    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8336388-5be6-4a25-8a3c-110e377e84da-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5jfwt\" (UID: \"c8336388-5be6-4a25-8a3c-110e377e84da\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt"
Sep 11 17:22:16.250432 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.250403    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8336388-5be6-4a25-8a3c-110e377e84da-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5jfwt\" (UID: \"c8336388-5be6-4a25-8a3c-110e377e84da\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt"
Sep 11 17:22:16.252219 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.252192    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:16.260836 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.260811    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lb4b\" (UniqueName: \"kubernetes.io/projected/c8336388-5be6-4a25-8a3c-110e377e84da-kube-api-access-9lb4b\") pod \"monitoring-plugin-5d85778b9d-5jfwt\" (UID: \"c8336388-5be6-4a25-8a3c-110e377e84da\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt"
Sep 11 17:22:16.349719 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.349644    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt"
Sep 11 17:22:16.515920 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.515885    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"]
Sep 11 17:22:16.516172 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.516147    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:16.518188 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.518165    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.521310 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.521289    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 17:22:16.521611 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.521596    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wc5rs\""
Sep 11 17:22:16.521916 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.521890    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 17:22:16.522119 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.522104    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 17:22:16.522338 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.522322    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 17:22:16.527950 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.527929    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 17:22:16.531626 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.531603    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 17:22:16.542215 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.542190    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"]
Sep 11 17:22:16.652154 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.652121    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c909851f-bc7f-4cba-a450-9a2fadba8e71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.652298 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.652189    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c909851f-bc7f-4cba-a450-9a2fadba8e71-serving-certs-ca-bundle\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.652298 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.652213    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c909851f-bc7f-4cba-a450-9a2fadba8e71-secret-telemeter-client\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.652298 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.652241    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c909851f-bc7f-4cba-a450-9a2fadba8e71-federate-client-tls\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.652298 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.652259    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c909851f-bc7f-4cba-a450-9a2fadba8e71-metrics-client-ca\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.652442 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.652370    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bgqc\" (UniqueName: \"kubernetes.io/projected/c909851f-bc7f-4cba-a450-9a2fadba8e71-kube-api-access-2bgqc\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.652442 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.652409    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c909851f-bc7f-4cba-a450-9a2fadba8e71-telemeter-client-tls\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.652527 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.652510    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c909851f-bc7f-4cba-a450-9a2fadba8e71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.685423 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.685396    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-647d54f9fd-s6g8m"]
Sep 11 17:22:16.725726 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.725693    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fd8f584b7-6dw5p"]
Sep 11 17:22:16.728537 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.728514    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.736584 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.736565    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 17:22:16.746691 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.746659    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd8f584b7-6dw5p"]
Sep 11 17:22:16.753050 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.753029    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c909851f-bc7f-4cba-a450-9a2fadba8e71-federate-client-tls\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.753050 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.753057    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c909851f-bc7f-4cba-a450-9a2fadba8e71-metrics-client-ca\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.753206 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.753120    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bgqc\" (UniqueName: \"kubernetes.io/projected/c909851f-bc7f-4cba-a450-9a2fadba8e71-kube-api-access-2bgqc\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.753206 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.753165    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c909851f-bc7f-4cba-a450-9a2fadba8e71-telemeter-client-tls\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.753297 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.753214    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c909851f-bc7f-4cba-a450-9a2fadba8e71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.753297 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.753241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c909851f-bc7f-4cba-a450-9a2fadba8e71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.753368 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.753316    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c909851f-bc7f-4cba-a450-9a2fadba8e71-serving-certs-ca-bundle\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.753368 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.753331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c909851f-bc7f-4cba-a450-9a2fadba8e71-secret-telemeter-client\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.755603 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.754223    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c909851f-bc7f-4cba-a450-9a2fadba8e71-metrics-client-ca\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.755603 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.755036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c909851f-bc7f-4cba-a450-9a2fadba8e71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.755603 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.755568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c909851f-bc7f-4cba-a450-9a2fadba8e71-serving-certs-ca-bundle\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.756301 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.756273    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c909851f-bc7f-4cba-a450-9a2fadba8e71-telemeter-client-tls\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.756619 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.756561    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c909851f-bc7f-4cba-a450-9a2fadba8e71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.756729 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.756709    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c909851f-bc7f-4cba-a450-9a2fadba8e71-secret-telemeter-client\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.758163 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.758143    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c909851f-bc7f-4cba-a450-9a2fadba8e71-federate-client-tls\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.768462 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.768441    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bgqc\" (UniqueName: \"kubernetes.io/projected/c909851f-bc7f-4cba-a450-9a2fadba8e71-kube-api-access-2bgqc\") pod \"telemeter-client-6f64d99d5b-tqhfc\" (UID: \"c909851f-bc7f-4cba-a450-9a2fadba8e71\") " pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.833557 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.833524    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"
Sep 11 17:22:16.854712 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.854685    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-service-ca\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.854834 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.854753    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa1b981c-cc06-4b5c-9987-25109d89df24-console-oauth-config\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.854904 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.854824    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa1b981c-cc06-4b5c-9987-25109d89df24-console-serving-cert\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.854904 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.854878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gn9m\" (UniqueName: \"kubernetes.io/projected/fa1b981c-cc06-4b5c-9987-25109d89df24-kube-api-access-8gn9m\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.855003 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.854905    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-console-config\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.855003 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.854928    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-trusted-ca-bundle\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.855003 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.854967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-oauth-serving-cert\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.956012 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.955928    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-oauth-serving-cert\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.956012 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.955994    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-service-ca\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.956255 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.956056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa1b981c-cc06-4b5c-9987-25109d89df24-console-oauth-config\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.956255 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.956087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa1b981c-cc06-4b5c-9987-25109d89df24-console-serving-cert\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.956255 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.956133    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gn9m\" (UniqueName: \"kubernetes.io/projected/fa1b981c-cc06-4b5c-9987-25109d89df24-kube-api-access-8gn9m\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.956255 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.956163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-console-config\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.956255 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.956193    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-trusted-ca-bundle\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.956849 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.956792    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-oauth-serving-cert\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.956958 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.956909    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-service-ca\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.957015 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.957000    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-trusted-ca-bundle\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.957123 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.957102    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-console-config\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.959528 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.959468    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa1b981c-cc06-4b5c-9987-25109d89df24-console-oauth-config\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.960251 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.960228    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa1b981c-cc06-4b5c-9987-25109d89df24-console-serving-cert\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:16.966778 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:16.966751    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gn9m\" (UniqueName: \"kubernetes.io/projected/fa1b981c-cc06-4b5c-9987-25109d89df24-kube-api-access-8gn9m\") pod \"console-fd8f584b7-6dw5p\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") " pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:17.040288 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.040066    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:17.227260 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:22:17.227068    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b337361_c434_4668_9ad2_a527b1789840.slice/crio-6f44847c1b011151021d36eb106278b7b8c61a18b818d444fe7d4f8e4b822c9e WatchSource:0}: Error finding container 6f44847c1b011151021d36eb106278b7b8c61a18b818d444fe7d4f8e4b822c9e: Status 404 returned error can't find the container with id 6f44847c1b011151021d36eb106278b7b8c61a18b818d444fe7d4f8e4b822c9e
Sep 11 17:22:17.505214 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.505147    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57c896d4fd-9hc4p"]
Sep 11 17:22:17.509865 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:22:17.509837    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76dde87e_bb30_4d65_9885_9fed9409ba76.slice/crio-2babda5abdb9b68811cdc5eabb1d50e9225e21c445c0aec484a4af60fb61af67 WatchSource:0}: Error finding container 2babda5abdb9b68811cdc5eabb1d50e9225e21c445c0aec484a4af60fb61af67: Status 404 returned error can't find the container with id 2babda5abdb9b68811cdc5eabb1d50e9225e21c445c0aec484a4af60fb61af67
Sep 11 17:22:17.538239 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.538195    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc"]
Sep 11 17:22:17.540933 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:22:17.540897    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ece35c6_9894_448b_9e3c_c90566b9d95e.slice/crio-1b4cb6e2d95cd4cc424eba1e2057441c9e73ac9828ec1316ea76e87f6f040b4c WatchSource:0}: Error finding container 1b4cb6e2d95cd4cc424eba1e2057441c9e73ac9828ec1316ea76e87f6f040b4c: Status 404 returned error can't find the container with id 1b4cb6e2d95cd4cc424eba1e2057441c9e73ac9828ec1316ea76e87f6f040b4c
Sep 11 17:22:17.558249 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.558224    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt"]
Sep 11 17:22:17.561744 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.561165    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd8f584b7-6dw5p"]
Sep 11 17:22:17.563994 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.563979    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:22:17.572115 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:22:17.572015    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76883eb1_c32d_47d4_9832_5f59dc8fcdfe.slice/crio-ac30c33014e9d2be77b908f49181b4ec3710d55ba0c866fd1f798bee3444e3a9 WatchSource:0}: Error finding container ac30c33014e9d2be77b908f49181b4ec3710d55ba0c866fd1f798bee3444e3a9: Status 404 returned error can't find the container with id ac30c33014e9d2be77b908f49181b4ec3710d55ba0c866fd1f798bee3444e3a9
Sep 11 17:22:17.583552 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.583525    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc"]
Sep 11 17:22:17.588596 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:22:17.588567    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc909851f_bc7f_4cba_a450_9a2fadba8e71.slice/crio-2e7db7aa98c701c9880e0a28be6f270b66a050acd56de3294865279a4f9f02ee WatchSource:0}: Error finding container 2e7db7aa98c701c9880e0a28be6f270b66a050acd56de3294865279a4f9f02ee: Status 404 returned error can't find the container with id 2e7db7aa98c701c9880e0a28be6f270b66a050acd56de3294865279a4f9f02ee
Sep 11 17:22:17.672407 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.672365    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd8f584b7-6dw5p" event={"ID":"fa1b981c-cc06-4b5c-9987-25109d89df24","Type":"ContainerStarted","Data":"8ac4045f0d10aa2da4aa6ae7f788cf44e5d857bb1b0e0fc5e9f33050c16e373e"}
Sep 11 17:22:17.672607 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.672432    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd8f584b7-6dw5p" event={"ID":"fa1b981c-cc06-4b5c-9987-25109d89df24","Type":"ContainerStarted","Data":"4e070b9fd2c0f30e1765d8b09654fc687b68c854a32a3a66463058013b5ea23a"}
Sep 11 17:22:17.674346 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.674298    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfbv6" event={"ID":"4b337361-c434-4668-9ad2-a527b1789840","Type":"ContainerStarted","Data":"6f44847c1b011151021d36eb106278b7b8c61a18b818d444fe7d4f8e4b822c9e"}
Sep 11 17:22:17.677720 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.677662    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-rbnrs" event={"ID":"54ac5c72-1b5d-4813-a4c7-6f6f108752a2","Type":"ContainerStarted","Data":"3151087e0838aebd48d9d6b19f8f0b98855152d8996cda95c8b1bde36d8e766e"}
Sep 11 17:22:17.678018 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.677974    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-rbnrs"
Sep 11 17:22:17.681670 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.681588    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p" event={"ID":"76dde87e-bb30-4d65-9885-9fed9409ba76","Type":"ContainerStarted","Data":"2babda5abdb9b68811cdc5eabb1d50e9225e21c445c0aec484a4af60fb61af67"}
Sep 11 17:22:17.684244 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.684157    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc" event={"ID":"c909851f-bc7f-4cba-a450-9a2fadba8e71","Type":"ContainerStarted","Data":"2e7db7aa98c701c9880e0a28be6f270b66a050acd56de3294865279a4f9f02ee"}
Sep 11 17:22:17.686399 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.686274    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc" event={"ID":"2ece35c6-9894-448b-9e3c-c90566b9d95e","Type":"ContainerStarted","Data":"1b4cb6e2d95cd4cc424eba1e2057441c9e73ac9828ec1316ea76e87f6f040b4c"}
Sep 11 17:22:17.689443 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.689369    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt" event={"ID":"c8336388-5be6-4a25-8a3c-110e377e84da","Type":"ContainerStarted","Data":"7c19c3f6f5d2f1f6cb8984008b70353fa1f1554f20d6fa0b3b6b9a968a32532e"}
Sep 11 17:22:17.691301 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.690607    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-rbnrs"
Sep 11 17:22:17.698173 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.697467    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x2zzk" event={"ID":"63bddd62-4518-490e-863d-f2d592e79ec3","Type":"ContainerStarted","Data":"63849976b1b0613a8868f726d77053c1a629a601dc8257e8a53db50808228f80"}
Sep 11 17:22:17.698173 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.697535    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x2zzk" event={"ID":"63bddd62-4518-490e-863d-f2d592e79ec3","Type":"ContainerStarted","Data":"ef28a97f111a9fb52329b123cc6085805b24154fbd91e5dbbe50799efadeb84b"}
Sep 11 17:22:17.698173 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.698113    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:22:17.700119 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.700035    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76883eb1-c32d-47d4-9832-5f59dc8fcdfe","Type":"ContainerStarted","Data":"ac30c33014e9d2be77b908f49181b4ec3710d55ba0c866fd1f798bee3444e3a9"}
Sep 11 17:22:17.729729 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.728994    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fd8f584b7-6dw5p" podStartSLOduration=1.728970222 podStartE2EDuration="1.728970222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:17.718208112 +0000 UTC m=+108.150660891" watchObservedRunningTime="2026-09-11 17:22:17.728970222 +0000 UTC m=+108.161423000"
Sep 11 17:22:17.748531 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.748437    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:17.863955 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.863796    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-rbnrs" podStartSLOduration=1.650418489 podStartE2EDuration="19.863771252s" totalImagesPullingTime="18.213352763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:17.798856866 +0000 UTC m=+108.231309666" watchObservedRunningTime="2026-09-11 17:22:17.863771252 +0000 UTC m=+108.296224028"
Sep 11 17:22:17.866669 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:17.864686    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x2zzk" podStartSLOduration=65.742641599 podStartE2EDuration="1m14.864671916s" totalImagesPullingTime="9.122030317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:17.863397362 +0000 UTC m=+108.295850141" watchObservedRunningTime="2026-09-11 17:22:17.864671916 +0000 UTC m=+108.297124695"
Sep 11 17:22:18.265316 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:18.265241    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:18.419323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:18.419225    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:18.806904 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:18.806866    2560 patch_prober.go:28] interesting pod/image-registry-86565b78d5-ngh5h container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 17:22:18.807402 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:18.806930    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-86565b78d5-ngh5h" podUID="5f1440e1-fa51-49ec-9ec8-686d89bcc606" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 17:22:20.594352 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:20.594059    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86565b78d5-ngh5h"
Sep 11 17:22:20.818237 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:20.818185    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:20.913897 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:20.913806    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:21.516163 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:21.516075    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:21.636899 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:21.636850    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:21.836257 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:21.836156    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:21.938075 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:21.938040    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:23.344977 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:23.344947    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:23.437257 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:23.437222    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:24.759825 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:24.758932    2560 generic.go:356] "Generic (PLEG): container finished" podID="4b337361-c434-4668-9ad2-a527b1789840" containerID="fea6cdd2641fa69c8e576d5a39e012b65e5210fa374c7491f3bc448dc48e36f9" exitCode=0
Sep 11 17:22:24.759825 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:24.759029    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfbv6" event={"ID":"4b337361-c434-4668-9ad2-a527b1789840","Type":"ContainerDied","Data":"fea6cdd2641fa69c8e576d5a39e012b65e5210fa374c7491f3bc448dc48e36f9"}
Sep 11 17:22:24.763914 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:24.763741    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bc548f8d-pbfmb" event={"ID":"a14a2cba-dc61-48ec-a462-4f79c6ff8858","Type":"ContainerStarted","Data":"c6536e7ccf1dbfb603c06ac434441effd3391d6d39ad694941f885f2c91915e0"}
Sep 11 17:22:24.766195 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:24.766135    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p" event={"ID":"76dde87e-bb30-4d65-9885-9fed9409ba76","Type":"ContainerStarted","Data":"81b2e7645bd68f766a60f22e95c1cb22cb1b589cbefa42ac7ad3ef2c9c5522ea"}
Sep 11 17:22:24.768848 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:24.768771    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc" event={"ID":"c909851f-bc7f-4cba-a450-9a2fadba8e71","Type":"ContainerStarted","Data":"8b4793a7ceec32b935769f6c1a3a497f0e1a988bb4b28c198fe028c411c9d70f"}
Sep 11 17:22:24.770905 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:24.770826    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc" event={"ID":"2ece35c6-9894-448b-9e3c-c90566b9d95e","Type":"ContainerStarted","Data":"6de2307b957083ac8f39e7e642b9b415da5be4cb96326f9e2e0e65f225097d06"}
Sep 11 17:22:24.773260 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:24.773055    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt" event={"ID":"c8336388-5be6-4a25-8a3c-110e377e84da","Type":"ContainerStarted","Data":"702aea99be21e5f6ac45d242a4e5c2e586d1bb815952e70b1659a6e9d528c9ab"}
Sep 11 17:22:24.776066 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:24.774450    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt"
Sep 11 17:22:24.781298 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:24.780567    2560 generic.go:356] "Generic (PLEG): container finished" podID="76883eb1-c32d-47d4-9832-5f59dc8fcdfe" containerID="af29acb5a124c5db3af142146ffc5820eebf9fc66bc5200d5224d884ede2b53c" exitCode=0
Sep 11 17:22:24.781298 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:24.780613    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76883eb1-c32d-47d4-9832-5f59dc8fcdfe","Type":"ContainerDied","Data":"af29acb5a124c5db3af142146ffc5820eebf9fc66bc5200d5224d884ede2b53c"}
Sep 11 17:22:24.786578 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:24.786529    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt"
Sep 11 17:22:24.831883 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:24.831125    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bc548f8d-pbfmb" podStartSLOduration=1.580227463 podStartE2EDuration="26.831102534s" totalImagesPullingTime="25.250875071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:21:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:24.805458209 +0000 UTC m=+115.237910990" watchObservedRunningTime="2026-09-11 17:22:24.831102534 +0000 UTC m=+115.263555310"
Sep 11 17:22:24.871059 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:24.867668    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p" podStartSLOduration=3.151755713 podStartE2EDuration="9.867646212s" totalImagesPullingTime="6.715890499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:24.833168209 +0000 UTC m=+115.265620990" watchObservedRunningTime="2026-09-11 17:22:24.867646212 +0000 UTC m=+115.300098989"
Sep 11 17:22:24.888534 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:24.887807    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5jfwt" podStartSLOduration=1.869074641 podStartE2EDuration="8.887784994s" totalImagesPullingTime="7.018710353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:24.886189819 +0000 UTC m=+115.318642894" watchObservedRunningTime="2026-09-11 17:22:24.887784994 +0000 UTC m=+115.320237774"
Sep 11 17:22:25.493091 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:25.493038    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:25.592759 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:25.592720    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:25.788917 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:25.788829    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc" event={"ID":"c909851f-bc7f-4cba-a450-9a2fadba8e71","Type":"ContainerStarted","Data":"7119b50369dfbe168e51fc3ff275b32a0f06927a433cd0023012fb3a15022c5b"}
Sep 11 17:22:25.789495 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:25.788927    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc" event={"ID":"c909851f-bc7f-4cba-a450-9a2fadba8e71","Type":"ContainerStarted","Data":"91fc47f5075fcaca584ed3cf112afaf385b87a5ae543c64f97e87fe0947fa9ec"}
Sep 11 17:22:25.791620 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:25.791588    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc" event={"ID":"2ece35c6-9894-448b-9e3c-c90566b9d95e","Type":"ContainerStarted","Data":"b6aa716f8bfb906597d60895df9a7a4c9bbf8bfd2c8c3cbe148db454f012d05b"}
Sep 11 17:22:25.791748 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:25.791644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc" event={"ID":"2ece35c6-9894-448b-9e3c-c90566b9d95e","Type":"ContainerStarted","Data":"4d9d2947e4c740950c10c0ac7be4a7fb6be65d3e740557970e1a79067790dc0d"}
Sep 11 17:22:25.794211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:25.794183    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfbv6" event={"ID":"4b337361-c434-4668-9ad2-a527b1789840","Type":"ContainerStarted","Data":"36f4f2eb6c88d87acf5b24cf6d557bf7fac93dc6bca7b8ac273bcd875d03c682"}
Sep 11 17:22:25.794313 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:25.794232    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfbv6" event={"ID":"4b337361-c434-4668-9ad2-a527b1789840","Type":"ContainerStarted","Data":"310ea7d8d79626345e4bcb281fa41a19f4820537e1dc6be6ad5e1ca591e4f5a5"}
Sep 11 17:22:25.814954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:25.814889    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f64d99d5b-tqhfc" podStartSLOduration=3.181596194 podStartE2EDuration="9.814870969s" totalImagesPullingTime="6.633274775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:25.814509577 +0000 UTC m=+116.246962359" watchObservedRunningTime="2026-09-11 17:22:25.814870969 +0000 UTC m=+116.247323747"
Sep 11 17:22:25.839331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:25.839286    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k4jzc" podStartSLOduration=8.158342422 podStartE2EDuration="14.839272518s" totalImagesPullingTime="6.680930096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:25.837075124 +0000 UTC m=+116.269527904" watchObservedRunningTime="2026-09-11 17:22:25.839272518 +0000 UTC m=+116.271725295"
Sep 11 17:22:25.857095 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:25.857032    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rfbv6" podStartSLOduration=7.765146481 podStartE2EDuration="14.857013503s" totalImagesPullingTime="6.995936466s" totalInitContainerRuntime="95.930556ms" isStatefulPod=true podCreationTimestamp="2026-09-11 17:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:25.856161761 +0000 UTC m=+116.288614541" watchObservedRunningTime="2026-09-11 17:22:25.857013503 +0000 UTC m=+116.289466281"
Sep 11 17:22:25.917198 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:25.917166    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fd8f584b7-6dw5p"]
Sep 11 17:22:26.555212 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:26.555155    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:26.631867 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:26.631833    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:27.040925 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:27.040885    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:27.809358 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:27.809266    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76883eb1-c32d-47d4-9832-5f59dc8fcdfe","Type":"ContainerStarted","Data":"bdb8317fff4d5a3d75143dee8e8751396f0691508d86f231da4ecafa56e35deb"}
Sep 11 17:22:27.809518 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:27.809393    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76883eb1-c32d-47d4-9832-5f59dc8fcdfe","Type":"ContainerStarted","Data":"ef3dee2baca7dd6d32ba4e2e4efe944f3802e4f780749e1f6f2195d3a1d55859"}
Sep 11 17:22:27.809518 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:27.809438    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76883eb1-c32d-47d4-9832-5f59dc8fcdfe","Type":"ContainerStarted","Data":"be470faa9af0f2b1f8842b9debea19bc3856ae97df5f55c27e2fff272e2b1ceb"}
Sep 11 17:22:27.809518 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:27.809492    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76883eb1-c32d-47d4-9832-5f59dc8fcdfe","Type":"ContainerStarted","Data":"7f5d7a66a1e7aa15d4249ce234e6152f40017cef15813e9546059b957af4cfa3"}
Sep 11 17:22:27.809635 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:27.809569    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76883eb1-c32d-47d4-9832-5f59dc8fcdfe","Type":"ContainerStarted","Data":"e7ea3a5505da3d1beb1de60db6409034b8ed26be02ae4686e6331e12bcb5da2c"}
Sep 11 17:22:28.721819 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:28.721789    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x2zzk"
Sep 11 17:22:29.256901 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:29.256821    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:29.388842 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:29.388810    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:29.511099 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:29.511031    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:29.580901 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:29.580869    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:29.821029 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:29.820945    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"76883eb1-c32d-47d4-9832-5f59dc8fcdfe","Type":"ContainerStarted","Data":"faa9bdde43d7b8f7feb70a8541e1adbc22ab896d140315d886628e7fdf987984"}
Sep 11 17:22:29.850913 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:29.850859    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=7.58257272 podStartE2EDuration="17.850836811s" totalImagesPullingTime="10.255792683s" totalInitContainerRuntime="12.471408ms" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:29.848849591 +0000 UTC m=+120.281302395" watchObservedRunningTime="2026-09-11 17:22:29.850836811 +0000 UTC m=+120.283289588"
Sep 11 17:22:36.031204 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:36.031167    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:36.031704 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:36.031259    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:36.704409 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:36.704371    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:36.802765 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:36.802733    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:37.532370 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:37.532336    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:37.651465 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:37.651430    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:38.629583 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:38.629545    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:38.742401 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:38.742372    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:38.866930 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:38.866895    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs\") pod \"network-metrics-daemon-8zghj\" (UID: \"d11d5fd7-89c6-4b5c-8340-9c75134f34bd\") " pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:22:38.869191 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:38.869165    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d11d5fd7-89c6-4b5c-8340-9c75134f34bd-metrics-certs\") pod \"network-metrics-daemon-8zghj\" (UID: \"d11d5fd7-89c6-4b5c-8340-9c75134f34bd\") " pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:22:38.964536 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:38.964503    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f9zbq\""
Sep 11 17:22:38.971756 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:38.971734    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8zghj"
Sep 11 17:22:39.279460 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:39.279439    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8zghj"]
Sep 11 17:22:39.281644 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:22:39.281618    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11d5fd7_89c6_4b5c_8340_9c75134f34bd.slice/crio-36b1a08fb7161cf4550a71853cf1b1d09d5a197ca461e1de7434e68cd9fbaee8 WatchSource:0}: Error finding container 36b1a08fb7161cf4550a71853cf1b1d09d5a197ca461e1de7434e68cd9fbaee8: Status 404 returned error can't find the container with id 36b1a08fb7161cf4550a71853cf1b1d09d5a197ca461e1de7434e68cd9fbaee8
Sep 11 17:22:39.862325 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:39.862289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8zghj" event={"ID":"d11d5fd7-89c6-4b5c-8340-9c75134f34bd","Type":"ContainerStarted","Data":"36b1a08fb7161cf4550a71853cf1b1d09d5a197ca461e1de7434e68cd9fbaee8"}
Sep 11 17:22:40.869323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:40.869289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8zghj" event={"ID":"d11d5fd7-89c6-4b5c-8340-9c75134f34bd","Type":"ContainerStarted","Data":"0f8737d52faa3966234cb1d9ba3c294e3d100e716ce2aa2874378db3aa775de9"}
Sep 11 17:22:41.705944 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:41.705880    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-647d54f9fd-s6g8m" podUID="4504947f-47bd-4fc2-b576-d43519a55ef6" containerName="console" containerID="cri-o://930f68cc39d928a02a4df7172a6225762e24d72a9047e93e8479ff240ae67eac" gracePeriod=15
Sep 11 17:22:41.876110 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:41.876070    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8zghj" event={"ID":"d11d5fd7-89c6-4b5c-8340-9c75134f34bd","Type":"ContainerStarted","Data":"054f0ec3950a48d3b441c06ffda9bfaf9c53c0a7e9f420305450988a5976eb6f"}
Sep 11 17:22:41.879232 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:41.879206    2560 generic.go:356] "Generic (PLEG): container finished" podID="4504947f-47bd-4fc2-b576-d43519a55ef6" containerID="930f68cc39d928a02a4df7172a6225762e24d72a9047e93e8479ff240ae67eac" exitCode=0
Sep 11 17:22:41.879363 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:41.879277    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647d54f9fd-s6g8m" event={"ID":"4504947f-47bd-4fc2-b576-d43519a55ef6","Type":"ContainerDied","Data":"930f68cc39d928a02a4df7172a6225762e24d72a9047e93e8479ff240ae67eac"}
Sep 11 17:22:41.901367 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:41.901284    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8zghj" podStartSLOduration=130.500526415 podStartE2EDuration="2m11.901261237s" totalImagesPullingTime="1.400734822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:41.898608875 +0000 UTC m=+132.331061654" watchObservedRunningTime="2026-09-11 17:22:41.901261237 +0000 UTC m=+132.333714015"
Sep 11 17:22:41.998668 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:41.998644    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:42.043109 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.043075    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:42.094209 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.094169    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-oauth-serving-cert\") pod \"4504947f-47bd-4fc2-b576-d43519a55ef6\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") "
Sep 11 17:22:42.094369 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.094258    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-service-ca\") pod \"4504947f-47bd-4fc2-b576-d43519a55ef6\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") "
Sep 11 17:22:42.094369 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.094282    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-console-config\") pod \"4504947f-47bd-4fc2-b576-d43519a55ef6\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") "
Sep 11 17:22:42.094369 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.094310    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4504947f-47bd-4fc2-b576-d43519a55ef6-console-oauth-config\") pod \"4504947f-47bd-4fc2-b576-d43519a55ef6\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") "
Sep 11 17:22:42.094539 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.094371    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc2nv\" (UniqueName: \"kubernetes.io/projected/4504947f-47bd-4fc2-b576-d43519a55ef6-kube-api-access-wc2nv\") pod \"4504947f-47bd-4fc2-b576-d43519a55ef6\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") "
Sep 11 17:22:42.094539 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.094416    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4504947f-47bd-4fc2-b576-d43519a55ef6-console-serving-cert\") pod \"4504947f-47bd-4fc2-b576-d43519a55ef6\" (UID: \"4504947f-47bd-4fc2-b576-d43519a55ef6\") "
Sep 11 17:22:42.094650 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.094612    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4504947f-47bd-4fc2-b576-d43519a55ef6" (UID: "4504947f-47bd-4fc2-b576-d43519a55ef6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:22:42.094710 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.094678    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-console-config" (OuterVolumeSpecName: "console-config") pod "4504947f-47bd-4fc2-b576-d43519a55ef6" (UID: "4504947f-47bd-4fc2-b576-d43519a55ef6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:22:42.094710 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.094688    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-service-ca" (OuterVolumeSpecName: "service-ca") pod "4504947f-47bd-4fc2-b576-d43519a55ef6" (UID: "4504947f-47bd-4fc2-b576-d43519a55ef6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:22:42.094815 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.094735    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-oauth-serving-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:22:42.096827 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.096788    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4504947f-47bd-4fc2-b576-d43519a55ef6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4504947f-47bd-4fc2-b576-d43519a55ef6" (UID: "4504947f-47bd-4fc2-b576-d43519a55ef6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:22:42.096931 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.096821    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4504947f-47bd-4fc2-b576-d43519a55ef6-kube-api-access-wc2nv" (OuterVolumeSpecName: "kube-api-access-wc2nv") pod "4504947f-47bd-4fc2-b576-d43519a55ef6" (UID: "4504947f-47bd-4fc2-b576-d43519a55ef6"). InnerVolumeSpecName "kube-api-access-wc2nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:22:42.096931 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.096852    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4504947f-47bd-4fc2-b576-d43519a55ef6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4504947f-47bd-4fc2-b576-d43519a55ef6" (UID: "4504947f-47bd-4fc2-b576-d43519a55ef6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:22:42.128837 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.128805    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:42.195864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.195823    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc2nv\" (UniqueName: \"kubernetes.io/projected/4504947f-47bd-4fc2-b576-d43519a55ef6-kube-api-access-wc2nv\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:22:42.195864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.195855    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4504947f-47bd-4fc2-b576-d43519a55ef6-console-serving-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:22:42.195864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.195868    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:22:42.196090 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.195882    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4504947f-47bd-4fc2-b576-d43519a55ef6-console-config\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:22:42.196090 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.195893    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4504947f-47bd-4fc2-b576-d43519a55ef6-console-oauth-config\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:22:42.204886 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.204861    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:42.883995 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.883964    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647d54f9fd-s6g8m"
Sep 11 17:22:42.884528 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.883995    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647d54f9fd-s6g8m" event={"ID":"4504947f-47bd-4fc2-b576-d43519a55ef6","Type":"ContainerDied","Data":"e0faa7d6512ecb001992999fdb6b790b98d4af7a979f2a533016f68ebb616ce0"}
Sep 11 17:22:42.884528 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.884052    2560 scope.go:117] "RemoveContainer" containerID="930f68cc39d928a02a4df7172a6225762e24d72a9047e93e8479ff240ae67eac"
Sep 11 17:22:42.905615 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.905589    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-647d54f9fd-s6g8m"]
Sep 11 17:22:42.909091 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:42.909070    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-647d54f9fd-s6g8m"]
Sep 11 17:22:44.049574 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:44.049538    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4504947f-47bd-4fc2-b576-d43519a55ef6" path="/var/lib/kubelet/pods/4504947f-47bd-4fc2-b576-d43519a55ef6/volumes"
Sep 11 17:22:44.491496 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:44.491445    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:44.554141 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:44.554112    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:45.548927 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:45.548895    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:45.650252 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:45.650224    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:45.727780 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:45.727746    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:50.937274 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:50.937212    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fd8f584b7-6dw5p" podUID="fa1b981c-cc06-4b5c-9987-25109d89df24" containerName="console" containerID="cri-o://8ac4045f0d10aa2da4aa6ae7f788cf44e5d857bb1b0e0fc5e9f33050c16e373e" gracePeriod=15
Sep 11 17:22:51.205597 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.205536    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:51.377582 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.377537    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa1b981c-cc06-4b5c-9987-25109d89df24-console-oauth-config\") pod \"fa1b981c-cc06-4b5c-9987-25109d89df24\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") "
Sep 11 17:22:51.377749 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.377603    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-trusted-ca-bundle\") pod \"fa1b981c-cc06-4b5c-9987-25109d89df24\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") "
Sep 11 17:22:51.377749 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.377648    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-service-ca\") pod \"fa1b981c-cc06-4b5c-9987-25109d89df24\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") "
Sep 11 17:22:51.377749 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.377669    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-oauth-serving-cert\") pod \"fa1b981c-cc06-4b5c-9987-25109d89df24\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") "
Sep 11 17:22:51.377749 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.377694    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gn9m\" (UniqueName: \"kubernetes.io/projected/fa1b981c-cc06-4b5c-9987-25109d89df24-kube-api-access-8gn9m\") pod \"fa1b981c-cc06-4b5c-9987-25109d89df24\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") "
Sep 11 17:22:51.377749 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.377735    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa1b981c-cc06-4b5c-9987-25109d89df24-console-serving-cert\") pod \"fa1b981c-cc06-4b5c-9987-25109d89df24\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") "
Sep 11 17:22:51.377977 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.377779    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-console-config\") pod \"fa1b981c-cc06-4b5c-9987-25109d89df24\" (UID: \"fa1b981c-cc06-4b5c-9987-25109d89df24\") "
Sep 11 17:22:51.378082 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.378055    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-service-ca" (OuterVolumeSpecName: "service-ca") pod "fa1b981c-cc06-4b5c-9987-25109d89df24" (UID: "fa1b981c-cc06-4b5c-9987-25109d89df24"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:22:51.378149 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.378090    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fa1b981c-cc06-4b5c-9987-25109d89df24" (UID: "fa1b981c-cc06-4b5c-9987-25109d89df24"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:22:51.378149 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.378097    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fa1b981c-cc06-4b5c-9987-25109d89df24" (UID: "fa1b981c-cc06-4b5c-9987-25109d89df24"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:22:51.378303 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.378285    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-console-config" (OuterVolumeSpecName: "console-config") pod "fa1b981c-cc06-4b5c-9987-25109d89df24" (UID: "fa1b981c-cc06-4b5c-9987-25109d89df24"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:22:51.379844 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.379808    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1b981c-cc06-4b5c-9987-25109d89df24-kube-api-access-8gn9m" (OuterVolumeSpecName: "kube-api-access-8gn9m") pod "fa1b981c-cc06-4b5c-9987-25109d89df24" (UID: "fa1b981c-cc06-4b5c-9987-25109d89df24"). InnerVolumeSpecName "kube-api-access-8gn9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:22:51.379988 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.379856    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa1b981c-cc06-4b5c-9987-25109d89df24-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fa1b981c-cc06-4b5c-9987-25109d89df24" (UID: "fa1b981c-cc06-4b5c-9987-25109d89df24"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:22:51.379988 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.379917    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa1b981c-cc06-4b5c-9987-25109d89df24-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fa1b981c-cc06-4b5c-9987-25109d89df24" (UID: "fa1b981c-cc06-4b5c-9987-25109d89df24"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:22:51.478407 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.478381    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa1b981c-cc06-4b5c-9987-25109d89df24-console-oauth-config\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:22:51.478407 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.478404    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-trusted-ca-bundle\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:22:51.478569 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.478413    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:22:51.478569 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.478423    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-oauth-serving-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:22:51.478569 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.478432    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gn9m\" (UniqueName: \"kubernetes.io/projected/fa1b981c-cc06-4b5c-9987-25109d89df24-kube-api-access-8gn9m\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:22:51.478569 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.478440    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa1b981c-cc06-4b5c-9987-25109d89df24-console-serving-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:22:51.478569 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.478449    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa1b981c-cc06-4b5c-9987-25109d89df24-console-config\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:22:51.925521 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.925461    2560 generic.go:356] "Generic (PLEG): container finished" podID="fa1b981c-cc06-4b5c-9987-25109d89df24" containerID="8ac4045f0d10aa2da4aa6ae7f788cf44e5d857bb1b0e0fc5e9f33050c16e373e" exitCode=0
Sep 11 17:22:51.925711 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.925529    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd8f584b7-6dw5p"
Sep 11 17:22:51.925711 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.925550    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd8f584b7-6dw5p" event={"ID":"fa1b981c-cc06-4b5c-9987-25109d89df24","Type":"ContainerDied","Data":"8ac4045f0d10aa2da4aa6ae7f788cf44e5d857bb1b0e0fc5e9f33050c16e373e"}
Sep 11 17:22:51.925711 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.925609    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd8f584b7-6dw5p" event={"ID":"fa1b981c-cc06-4b5c-9987-25109d89df24","Type":"ContainerDied","Data":"4e070b9fd2c0f30e1765d8b09654fc687b68c854a32a3a66463058013b5ea23a"}
Sep 11 17:22:51.925711 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.925640    2560 scope.go:117] "RemoveContainer" containerID="8ac4045f0d10aa2da4aa6ae7f788cf44e5d857bb1b0e0fc5e9f33050c16e373e"
Sep 11 17:22:51.927302 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.927278    2560 generic.go:356] "Generic (PLEG): container finished" podID="e353cc34-4593-4fa9-bba7-6509ecef06cc" containerID="578cb2e432f4f74de491d1f58a7768283b15bda5eaafa18296ff443359e04d20" exitCode=0
Sep 11 17:22:51.927391 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.927348    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-24qlf" event={"ID":"e353cc34-4593-4fa9-bba7-6509ecef06cc","Type":"ContainerDied","Data":"578cb2e432f4f74de491d1f58a7768283b15bda5eaafa18296ff443359e04d20"}
Sep 11 17:22:51.927670 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.927657    2560 scope.go:117] "RemoveContainer" containerID="578cb2e432f4f74de491d1f58a7768283b15bda5eaafa18296ff443359e04d20"
Sep 11 17:22:51.934626 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.934611    2560 scope.go:117] "RemoveContainer" containerID="8ac4045f0d10aa2da4aa6ae7f788cf44e5d857bb1b0e0fc5e9f33050c16e373e"
Sep 11 17:22:51.934938 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:22:51.934912    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac4045f0d10aa2da4aa6ae7f788cf44e5d857bb1b0e0fc5e9f33050c16e373e\": container with ID starting with 8ac4045f0d10aa2da4aa6ae7f788cf44e5d857bb1b0e0fc5e9f33050c16e373e not found: ID does not exist" containerID="8ac4045f0d10aa2da4aa6ae7f788cf44e5d857bb1b0e0fc5e9f33050c16e373e"
Sep 11 17:22:51.935022 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.934946    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac4045f0d10aa2da4aa6ae7f788cf44e5d857bb1b0e0fc5e9f33050c16e373e"} err="failed to get container status \"8ac4045f0d10aa2da4aa6ae7f788cf44e5d857bb1b0e0fc5e9f33050c16e373e\": rpc error: code = NotFound desc = could not find container \"8ac4045f0d10aa2da4aa6ae7f788cf44e5d857bb1b0e0fc5e9f33050c16e373e\": container with ID starting with 8ac4045f0d10aa2da4aa6ae7f788cf44e5d857bb1b0e0fc5e9f33050c16e373e not found: ID does not exist"
Sep 11 17:22:51.970044 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.970015    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fd8f584b7-6dw5p"]
Sep 11 17:22:51.974459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:51.974433    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fd8f584b7-6dw5p"]
Sep 11 17:22:52.050712 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:52.050687    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa1b981c-cc06-4b5c-9987-25109d89df24" path="/var/lib/kubelet/pods/fa1b981c-cc06-4b5c-9987-25109d89df24/volumes"
Sep 11 17:22:52.933853 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:52.933822    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-24qlf" event={"ID":"e353cc34-4593-4fa9-bba7-6509ecef06cc","Type":"ContainerStarted","Data":"af209c4ec4edeb3e30933ebc4307481c7b83ac1dc6aff8c3fb91ef1b9cdc2805"}
Sep 11 17:22:56.036218 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:56.036189    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:56.039897 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:56.039873    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-57c896d4fd-9hc4p"
Sep 11 17:22:59.510519 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:59.510487    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:22:59.614467 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:22:59.614437    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:23:12.451418 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:12.451382    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:23:27.542246 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:27.542145    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:23:27.642166 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:27.642132    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:23:37.746401 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:37.746337    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:23:39.437879 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:39.437836    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:23:39.501553 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:39.501461    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:23:43.450598 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.450566    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b5d5b8444-7v7vt"]
Sep 11 17:23:43.450960 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.450859    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4504947f-47bd-4fc2-b576-d43519a55ef6" containerName="console"
Sep 11 17:23:43.450960 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.450869    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4504947f-47bd-4fc2-b576-d43519a55ef6" containerName="console"
Sep 11 17:23:43.450960 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.450890    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa1b981c-cc06-4b5c-9987-25109d89df24" containerName="console"
Sep 11 17:23:43.450960 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.450895    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa1b981c-cc06-4b5c-9987-25109d89df24" containerName="console"
Sep 11 17:23:43.450960 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.450939    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4504947f-47bd-4fc2-b576-d43519a55ef6" containerName="console"
Sep 11 17:23:43.450960 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.450947    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa1b981c-cc06-4b5c-9987-25109d89df24" containerName="console"
Sep 11 17:23:43.453749 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.453731    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.457034 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.457006    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 17:23:43.457034 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.457022    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 17:23:43.457244 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.457228    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 17:23:43.457597 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.457574    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 17:23:43.457701 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.457581    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9tzbh\""
Sep 11 17:23:43.457760 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.457707    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 17:23:43.463754 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.463731    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 17:23:43.467314 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.467297    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b5d5b8444-7v7vt"]
Sep 11 17:23:43.511745 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.511715    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:23:43.575550 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.575525    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:23:43.589757 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.589734    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tpx7\" (UniqueName: \"kubernetes.io/projected/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-kube-api-access-7tpx7\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.589861 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.589808    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-trusted-ca-bundle\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.589861 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.589834    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-oauth-config\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.589861 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.589850    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-config\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.590015 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.589915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-serving-cert\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.590015 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.589947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-service-ca\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.590015 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.589981    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-oauth-serving-cert\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.690974 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.690940    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-trusted-ca-bundle\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.691086 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.690982    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-oauth-config\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.691086 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.691005    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-config\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.691086 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.691043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-serving-cert\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.691086 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.691065    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-service-ca\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.691273 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.691097    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-oauth-serving-cert\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.691273 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.691155    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tpx7\" (UniqueName: \"kubernetes.io/projected/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-kube-api-access-7tpx7\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.691971 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.691944    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-config\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.692140 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.691976    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-service-ca\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.692140 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.692016    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-oauth-serving-cert\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.692387 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.692365    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-trusted-ca-bundle\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.694140 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.694117    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-oauth-config\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.694214 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.694200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-serving-cert\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.699589 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.699567    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tpx7\" (UniqueName: \"kubernetes.io/projected/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-kube-api-access-7tpx7\") pod \"console-6b5d5b8444-7v7vt\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") " pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:43.764230 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:43.764162    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:44.033881 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:44.033812    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b5d5b8444-7v7vt"]
Sep 11 17:23:44.037134 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:23:44.037107    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020a3b4b_e11f_4cf4_9178_9b787de8ddcd.slice/crio-70f1e1db815a3e19d565000b0ae18bbe5ba98f75cd2b90c9ae47a58fbfcee498 WatchSource:0}: Error finding container 70f1e1db815a3e19d565000b0ae18bbe5ba98f75cd2b90c9ae47a58fbfcee498: Status 404 returned error can't find the container with id 70f1e1db815a3e19d565000b0ae18bbe5ba98f75cd2b90c9ae47a58fbfcee498
Sep 11 17:23:44.151876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:44.151840    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5d5b8444-7v7vt" event={"ID":"020a3b4b-e11f-4cf4-9178-9b787de8ddcd","Type":"ContainerStarted","Data":"a1d2e02df00b98482c2350070ec8edd60475a27130fee9cdc4c940980a0a66c3"}
Sep 11 17:23:44.152039 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:44.151894    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5d5b8444-7v7vt" event={"ID":"020a3b4b-e11f-4cf4-9178-9b787de8ddcd","Type":"ContainerStarted","Data":"70f1e1db815a3e19d565000b0ae18bbe5ba98f75cd2b90c9ae47a58fbfcee498"}
Sep 11 17:23:44.191747 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:44.191696    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b5d5b8444-7v7vt" podStartSLOduration=1.191681724 podStartE2EDuration="1.191681724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:23:44.191000714 +0000 UTC m=+194.623453489" watchObservedRunningTime="2026-09-11 17:23:44.191681724 +0000 UTC m=+194.624134501"
Sep 11 17:23:44.510890 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:44.510858    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:23:44.609256 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:44.609220    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:23:53.765032 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:53.764969    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:53.765434 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:53.765117    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:53.769756 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:53.769733    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:54.194321 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:54.194279    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:23:54.386833 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:54.386796    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:23:54.487217 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:54.487120    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:23:54.587196 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:23:54.587159    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:16.833584 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:16.833554    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:22.281604 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:22.281564    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:22.387025 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:22.386968    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:46.301228 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:46.301195    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:46.402568 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:46.402520    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:50.747716 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:50.747672    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:50.956620 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:50.956566    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:52.534048 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:52.533968    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:52.727388 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:52.727359    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:53.009483 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:53.009441    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:54.929042 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:54.928996    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:55.297032 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:55.296916    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:55.509770 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:55.509731    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-x6lf2"]
Sep 11 17:24:55.511726 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:55.511711    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-x6lf2"
Sep 11 17:24:55.514676 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:55.514656    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 17:24:55.515566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:55.515548    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 17:24:55.515669 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:55.515550    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qq8q7\""
Sep 11 17:24:55.528342 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:55.528323    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-x6lf2"]
Sep 11 17:24:55.597753 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:55.597678    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c49dda6b-2e13-43db-907d-8a56514f5314-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-x6lf2\" (UID: \"c49dda6b-2e13-43db-907d-8a56514f5314\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x6lf2"
Sep 11 17:24:55.597875 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:55.597794    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfbsn\" (UniqueName: \"kubernetes.io/projected/c49dda6b-2e13-43db-907d-8a56514f5314-kube-api-access-rfbsn\") pod \"cert-manager-cainjector-559cdd8c87-x6lf2\" (UID: \"c49dda6b-2e13-43db-907d-8a56514f5314\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x6lf2"
Sep 11 17:24:55.699146 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:55.699116    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfbsn\" (UniqueName: \"kubernetes.io/projected/c49dda6b-2e13-43db-907d-8a56514f5314-kube-api-access-rfbsn\") pod \"cert-manager-cainjector-559cdd8c87-x6lf2\" (UID: \"c49dda6b-2e13-43db-907d-8a56514f5314\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x6lf2"
Sep 11 17:24:55.699273 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:55.699156    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c49dda6b-2e13-43db-907d-8a56514f5314-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-x6lf2\" (UID: \"c49dda6b-2e13-43db-907d-8a56514f5314\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x6lf2"
Sep 11 17:24:55.711384 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:55.711351    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfbsn\" (UniqueName: \"kubernetes.io/projected/c49dda6b-2e13-43db-907d-8a56514f5314-kube-api-access-rfbsn\") pod \"cert-manager-cainjector-559cdd8c87-x6lf2\" (UID: \"c49dda6b-2e13-43db-907d-8a56514f5314\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x6lf2"
Sep 11 17:24:55.711969 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:55.711950    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c49dda6b-2e13-43db-907d-8a56514f5314-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-x6lf2\" (UID: \"c49dda6b-2e13-43db-907d-8a56514f5314\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-x6lf2"
Sep 11 17:24:55.820627 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:55.820606    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-x6lf2"
Sep 11 17:24:56.111988 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:56.111966    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-x6lf2"]
Sep 11 17:24:56.114497 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:24:56.114417    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc49dda6b_2e13_43db_907d_8a56514f5314.slice/crio-d0800a19b8519652913af55273ea1baf1e5a19826b316ae3ed8cd6d3bb6c0feb WatchSource:0}: Error finding container d0800a19b8519652913af55273ea1baf1e5a19826b316ae3ed8cd6d3bb6c0feb: Status 404 returned error can't find the container with id d0800a19b8519652913af55273ea1baf1e5a19826b316ae3ed8cd6d3bb6c0feb
Sep 11 17:24:56.430330 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:56.430294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-x6lf2" event={"ID":"c49dda6b-2e13-43db-907d-8a56514f5314","Type":"ContainerStarted","Data":"d0800a19b8519652913af55273ea1baf1e5a19826b316ae3ed8cd6d3bb6c0feb"}
Sep 11 17:24:56.684760 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:56.684690    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:56.783384 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:56.783339    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:24:59.444696 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:59.444663    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-x6lf2" event={"ID":"c49dda6b-2e13-43db-907d-8a56514f5314","Type":"ContainerStarted","Data":"18f1e7305338faf076fa7245dba1134812d3628288682f3a91fe8871813b7c2f"}
Sep 11 17:24:59.465464 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:24:59.465394    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-x6lf2" podStartSLOduration=1.635293505 podStartE2EDuration="4.46537718s" totalImagesPullingTime="2.830083675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:24:59.464875611 +0000 UTC m=+269.897328388" watchObservedRunningTime="2026-09-11 17:24:59.46537718 +0000 UTC m=+269.897829953"
Sep 11 17:25:03.576779 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:03.576738    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:06.989187 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:06.989155    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:07.075403 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:07.075371    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:13.731945 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:13.731895    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:13.825399 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:13.825369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:15.572946 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:15.572915    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:21.616583 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:21.616547    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:21.698504 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:21.698457    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:27.493604 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:27.493570    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:27.597511 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:27.597457    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:27.840406 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:27.840320    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:28.332983 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.332951    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:28.435824 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.435783    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:28.489987 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.489956    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"]
Sep 11 17:25:28.496646 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.496622    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.499788 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.499766    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 17:25:28.501099 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.501080    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 17:25:28.501206 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.501121    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 17:25:28.501206 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.501126    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hw64m\""
Sep 11 17:25:28.501206 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.501130    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 17:25:28.526331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.526300    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"]
Sep 11 17:25:28.574696 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.574663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xrnr\" (UniqueName: \"kubernetes.io/projected/5db69f6a-8a40-4538-9770-278a8df11739-kube-api-access-7xrnr\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-jbpht\" (UID: \"5db69f6a-8a40-4538-9770-278a8df11739\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.574844 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.574702    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5db69f6a-8a40-4538-9770-278a8df11739-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-jbpht\" (UID: \"5db69f6a-8a40-4538-9770-278a8df11739\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.574844 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.574726    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5db69f6a-8a40-4538-9770-278a8df11739-manifests\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-jbpht\" (UID: \"5db69f6a-8a40-4538-9770-278a8df11739\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.574844 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.574790    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5db69f6a-8a40-4538-9770-278a8df11739-tmp\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-jbpht\" (UID: \"5db69f6a-8a40-4538-9770-278a8df11739\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.574962 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.574846    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5db69f6a-8a40-4538-9770-278a8df11739-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-jbpht\" (UID: \"5db69f6a-8a40-4538-9770-278a8df11739\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.652732 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.652705    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:28.675893 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.675859    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xrnr\" (UniqueName: \"kubernetes.io/projected/5db69f6a-8a40-4538-9770-278a8df11739-kube-api-access-7xrnr\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-jbpht\" (UID: \"5db69f6a-8a40-4538-9770-278a8df11739\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.676023 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.675902    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5db69f6a-8a40-4538-9770-278a8df11739-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-jbpht\" (UID: \"5db69f6a-8a40-4538-9770-278a8df11739\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.676023 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.675929    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5db69f6a-8a40-4538-9770-278a8df11739-manifests\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-jbpht\" (UID: \"5db69f6a-8a40-4538-9770-278a8df11739\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.676023 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.675964    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5db69f6a-8a40-4538-9770-278a8df11739-tmp\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-jbpht\" (UID: \"5db69f6a-8a40-4538-9770-278a8df11739\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.676023 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.676000    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5db69f6a-8a40-4538-9770-278a8df11739-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-jbpht\" (UID: \"5db69f6a-8a40-4538-9770-278a8df11739\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.676348 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.676327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5db69f6a-8a40-4538-9770-278a8df11739-manifests\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-jbpht\" (UID: \"5db69f6a-8a40-4538-9770-278a8df11739\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.678319 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.678295    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5db69f6a-8a40-4538-9770-278a8df11739-tmp\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-jbpht\" (UID: \"5db69f6a-8a40-4538-9770-278a8df11739\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.678592 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.678570    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5db69f6a-8a40-4538-9770-278a8df11739-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-jbpht\" (UID: \"5db69f6a-8a40-4538-9770-278a8df11739\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.678644 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.678607    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5db69f6a-8a40-4538-9770-278a8df11739-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-jbpht\" (UID: \"5db69f6a-8a40-4538-9770-278a8df11739\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.686943 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.686918    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xrnr\" (UniqueName: \"kubernetes.io/projected/5db69f6a-8a40-4538-9770-278a8df11739-kube-api-access-7xrnr\") pod \"opendatahub-operator-controller-manager-7f9c58fc78-jbpht\" (UID: \"5db69f6a-8a40-4538-9770-278a8df11739\") " pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.751660 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.751632    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:28.807755 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.807731    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:28.852862 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.852782    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:28.960072 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:28.959973    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:29.090321 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:29.090288    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:29.134312 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:29.134291    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"]
Sep 11 17:25:29.135149 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:25:29.135127    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db69f6a_8a40_4538_9770_278a8df11739.slice/crio-480265e3c6b04713b9d477da1ed769ff1bd742b0c53b0ec14547b41bdf6d7eca WatchSource:0}: Error finding container 480265e3c6b04713b9d477da1ed769ff1bd742b0c53b0ec14547b41bdf6d7eca: Status 404 returned error can't find the container with id 480265e3c6b04713b9d477da1ed769ff1bd742b0c53b0ec14547b41bdf6d7eca
Sep 11 17:25:29.566988 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:29.566958    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht" event={"ID":"5db69f6a-8a40-4538-9770-278a8df11739","Type":"ContainerStarted","Data":"480265e3c6b04713b9d477da1ed769ff1bd742b0c53b0ec14547b41bdf6d7eca"}
Sep 11 17:25:29.952260 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:29.952177    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:25:29.952826 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:29.952802    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:25:29.962517 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:29.962493    2560 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 17:25:30.203012 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:30.202872    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:32.583231 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:32.583154    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht" event={"ID":"5db69f6a-8a40-4538-9770-278a8df11739","Type":"ContainerStarted","Data":"b668a25b570cf22c8b1a75f6b79187fa146e456757429c832c66f451f0a10ae9"}
Sep 11 17:25:33.589504 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:33.589446    2560 generic.go:356] "Generic (PLEG): container finished" podID="5db69f6a-8a40-4538-9770-278a8df11739" containerID="b668a25b570cf22c8b1a75f6b79187fa146e456757429c832c66f451f0a10ae9" exitCode=0
Sep 11 17:25:33.589866 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:33.589537    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht" event={"ID":"5db69f6a-8a40-4538-9770-278a8df11739","Type":"ContainerDied","Data":"b668a25b570cf22c8b1a75f6b79187fa146e456757429c832c66f451f0a10ae9"}
Sep 11 17:25:33.590120 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:33.590104    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:25:34.596058 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:34.596020    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht" event={"ID":"5db69f6a-8a40-4538-9770-278a8df11739","Type":"ContainerStarted","Data":"f582675ab9eeab0da8a755b0c3bb631f7cc5939d5f3fa4cac74895226ee87dac"}
Sep 11 17:25:34.596546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:34.596168    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:34.626131 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:34.626084    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht" podStartSLOduration=2.023709797 podStartE2EDuration="6.626067388s" totalImagesPullingTime="3.54616293s" totalInitContainerRuntime="1.056194661s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:25:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:25:34.624754816 +0000 UTC m=+305.057207593" watchObservedRunningTime="2026-09-11 17:25:34.626067388 +0000 UTC m=+305.058520163"
Sep 11 17:25:36.758352 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:36.758322    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:37.076195 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:37.076117    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:37.173729 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:37.173703    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:45.603758 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:45.603718    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f9c58fc78-jbpht"
Sep 11 17:25:47.784881 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:47.784846    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:47.937905 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:47.937874    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:58.596182 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.596145    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:25:58.812031 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.811994    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2"]
Sep 11 17:25:58.815559 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.815535    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2"
Sep 11 17:25:58.821855 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.821827    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jdklt\""
Sep 11 17:25:58.822309 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.822176    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 17:25:58.822683 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.822664    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 17:25:58.828486 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.828451    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9c38b6d-7101-4132-ae6e-377a97881082-tmp\") pod \"kube-auth-proxy-5897fcd6c7-qgbz2\" (UID: \"c9c38b6d-7101-4132-ae6e-377a97881082\") " pod="openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2"
Sep 11 17:25:58.828597 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.828532    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt79l\" (UniqueName: \"kubernetes.io/projected/c9c38b6d-7101-4132-ae6e-377a97881082-kube-api-access-bt79l\") pod \"kube-auth-proxy-5897fcd6c7-qgbz2\" (UID: \"c9c38b6d-7101-4132-ae6e-377a97881082\") " pod="openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2"
Sep 11 17:25:58.828597 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.828583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c38b6d-7101-4132-ae6e-377a97881082-tls-certs\") pod \"kube-auth-proxy-5897fcd6c7-qgbz2\" (UID: \"c9c38b6d-7101-4132-ae6e-377a97881082\") " pod="openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2"
Sep 11 17:25:58.832568 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.832542    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2"]
Sep 11 17:25:58.929653 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.929619    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9c38b6d-7101-4132-ae6e-377a97881082-tmp\") pod \"kube-auth-proxy-5897fcd6c7-qgbz2\" (UID: \"c9c38b6d-7101-4132-ae6e-377a97881082\") " pod="openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2"
Sep 11 17:25:58.929834 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.929674    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt79l\" (UniqueName: \"kubernetes.io/projected/c9c38b6d-7101-4132-ae6e-377a97881082-kube-api-access-bt79l\") pod \"kube-auth-proxy-5897fcd6c7-qgbz2\" (UID: \"c9c38b6d-7101-4132-ae6e-377a97881082\") " pod="openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2"
Sep 11 17:25:58.929834 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.929721    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c38b6d-7101-4132-ae6e-377a97881082-tls-certs\") pod \"kube-auth-proxy-5897fcd6c7-qgbz2\" (UID: \"c9c38b6d-7101-4132-ae6e-377a97881082\") " pod="openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2"
Sep 11 17:25:58.929917 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:25:58.929908    2560 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 11 17:25:58.929985 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:25:58.929974    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9c38b6d-7101-4132-ae6e-377a97881082-tls-certs podName:c9c38b6d-7101-4132-ae6e-377a97881082 nodeName:}" failed. No retries permitted until 2026-09-11 17:25:59.429954178 +0000 UTC m=+329.862406935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c9c38b6d-7101-4132-ae6e-377a97881082-tls-certs") pod "kube-auth-proxy-5897fcd6c7-qgbz2" (UID: "c9c38b6d-7101-4132-ae6e-377a97881082") : secret "kube-auth-proxy-tls" not found
Sep 11 17:25:58.931947 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.931910    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9c38b6d-7101-4132-ae6e-377a97881082-tmp\") pod \"kube-auth-proxy-5897fcd6c7-qgbz2\" (UID: \"c9c38b6d-7101-4132-ae6e-377a97881082\") " pod="openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2"
Sep 11 17:25:58.943847 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.943820    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-d5vf9"]
Sep 11 17:25:58.947195 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.947172    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-d5vf9"
Sep 11 17:25:58.950618 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.950596    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-cqspc\""
Sep 11 17:25:58.950845 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.950687    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 11 17:25:58.953181 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.953128    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt79l\" (UniqueName: \"kubernetes.io/projected/c9c38b6d-7101-4132-ae6e-377a97881082-kube-api-access-bt79l\") pod \"kube-auth-proxy-5897fcd6c7-qgbz2\" (UID: \"c9c38b6d-7101-4132-ae6e-377a97881082\") " pod="openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2"
Sep 11 17:25:58.961413 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:58.961387    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-d5vf9"]
Sep 11 17:25:59.030807 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:59.030761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e71d59dc-fde9-4176-afa5-8d6f015d6157-cert\") pod \"odh-model-controller-7998c67dc4-d5vf9\" (UID: \"e71d59dc-fde9-4176-afa5-8d6f015d6157\") " pod="opendatahub/odh-model-controller-7998c67dc4-d5vf9"
Sep 11 17:25:59.030807 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:59.030811    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7vnf\" (UniqueName: \"kubernetes.io/projected/e71d59dc-fde9-4176-afa5-8d6f015d6157-kube-api-access-c7vnf\") pod \"odh-model-controller-7998c67dc4-d5vf9\" (UID: \"e71d59dc-fde9-4176-afa5-8d6f015d6157\") " pod="opendatahub/odh-model-controller-7998c67dc4-d5vf9"
Sep 11 17:25:59.131415 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:59.131372    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e71d59dc-fde9-4176-afa5-8d6f015d6157-cert\") pod \"odh-model-controller-7998c67dc4-d5vf9\" (UID: \"e71d59dc-fde9-4176-afa5-8d6f015d6157\") " pod="opendatahub/odh-model-controller-7998c67dc4-d5vf9"
Sep 11 17:25:59.131612 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:59.131434    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7vnf\" (UniqueName: \"kubernetes.io/projected/e71d59dc-fde9-4176-afa5-8d6f015d6157-kube-api-access-c7vnf\") pod \"odh-model-controller-7998c67dc4-d5vf9\" (UID: \"e71d59dc-fde9-4176-afa5-8d6f015d6157\") " pod="opendatahub/odh-model-controller-7998c67dc4-d5vf9"
Sep 11 17:25:59.131612 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:25:59.131529    2560 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 11 17:25:59.131612 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:25:59.131595    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e71d59dc-fde9-4176-afa5-8d6f015d6157-cert podName:e71d59dc-fde9-4176-afa5-8d6f015d6157 nodeName:}" failed. No retries permitted until 2026-09-11 17:25:59.631579288 +0000 UTC m=+330.064032042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e71d59dc-fde9-4176-afa5-8d6f015d6157-cert") pod "odh-model-controller-7998c67dc4-d5vf9" (UID: "e71d59dc-fde9-4176-afa5-8d6f015d6157") : secret "odh-model-controller-webhook-cert" not found
Sep 11 17:25:59.145251 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:59.145220    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7vnf\" (UniqueName: \"kubernetes.io/projected/e71d59dc-fde9-4176-afa5-8d6f015d6157-kube-api-access-c7vnf\") pod \"odh-model-controller-7998c67dc4-d5vf9\" (UID: \"e71d59dc-fde9-4176-afa5-8d6f015d6157\") " pod="opendatahub/odh-model-controller-7998c67dc4-d5vf9"
Sep 11 17:25:59.434170 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:59.434133    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c38b6d-7101-4132-ae6e-377a97881082-tls-certs\") pod \"kube-auth-proxy-5897fcd6c7-qgbz2\" (UID: \"c9c38b6d-7101-4132-ae6e-377a97881082\") " pod="openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2"
Sep 11 17:25:59.436427 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:59.436395    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c38b6d-7101-4132-ae6e-377a97881082-tls-certs\") pod \"kube-auth-proxy-5897fcd6c7-qgbz2\" (UID: \"c9c38b6d-7101-4132-ae6e-377a97881082\") " pod="openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2"
Sep 11 17:25:59.635873 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:59.635837    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e71d59dc-fde9-4176-afa5-8d6f015d6157-cert\") pod \"odh-model-controller-7998c67dc4-d5vf9\" (UID: \"e71d59dc-fde9-4176-afa5-8d6f015d6157\") " pod="opendatahub/odh-model-controller-7998c67dc4-d5vf9"
Sep 11 17:25:59.638162 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:59.638131    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e71d59dc-fde9-4176-afa5-8d6f015d6157-cert\") pod \"odh-model-controller-7998c67dc4-d5vf9\" (UID: \"e71d59dc-fde9-4176-afa5-8d6f015d6157\") " pod="opendatahub/odh-model-controller-7998c67dc4-d5vf9"
Sep 11 17:25:59.725043 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:59.724954    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2"
Sep 11 17:25:59.866724 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:59.866680    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-d5vf9"
Sep 11 17:25:59.924088 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:25:59.923885    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:00.042512 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:00.042490    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2"]
Sep 11 17:26:00.046166 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:26:00.046113    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c38b6d_7101_4132_ae6e_377a97881082.slice/crio-f0df752733720f008d6e46e42b384b584eb45027a388865a5a20a1eea9ef1255 WatchSource:0}: Error finding container f0df752733720f008d6e46e42b384b584eb45027a388865a5a20a1eea9ef1255: Status 404 returned error can't find the container with id f0df752733720f008d6e46e42b384b584eb45027a388865a5a20a1eea9ef1255
Sep 11 17:26:00.090112 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:00.090079    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:00.148586 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:00.148512    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-d5vf9"]
Sep 11 17:26:00.153090 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:26:00.153053    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71d59dc_fde9_4176_afa5_8d6f015d6157.slice/crio-74623ee6734abd5898c78481f0351835e7084291548492f09277b7fc7bd8f499 WatchSource:0}: Error finding container 74623ee6734abd5898c78481f0351835e7084291548492f09277b7fc7bd8f499: Status 404 returned error can't find the container with id 74623ee6734abd5898c78481f0351835e7084291548492f09277b7fc7bd8f499
Sep 11 17:26:00.251811 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:00.251775    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:00.335993 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:00.335904    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:00.724549 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:00.717875    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2" event={"ID":"c9c38b6d-7101-4132-ae6e-377a97881082","Type":"ContainerStarted","Data":"f0df752733720f008d6e46e42b384b584eb45027a388865a5a20a1eea9ef1255"}
Sep 11 17:26:00.724549 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:00.722168    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-d5vf9" event={"ID":"e71d59dc-fde9-4176-afa5-8d6f015d6157","Type":"ContainerStarted","Data":"74623ee6734abd5898c78481f0351835e7084291548492f09277b7fc7bd8f499"}
Sep 11 17:26:01.705910 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:01.705866    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:04.746706 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:04.746670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-d5vf9" event={"ID":"e71d59dc-fde9-4176-afa5-8d6f015d6157","Type":"ContainerStarted","Data":"e0cf30c74f4410718dda6405489e0b9207b3168eb0878baa17e5f8e51a254ad4"}
Sep 11 17:26:04.747108 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:04.746790    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-d5vf9"
Sep 11 17:26:04.769566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:04.769415    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-d5vf9" podStartSLOduration=2.392030868 podStartE2EDuration="6.769395074s" totalImagesPullingTime="4.377364206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:26:04.769337173 +0000 UTC m=+335.201789950" watchObservedRunningTime="2026-09-11 17:26:04.769395074 +0000 UTC m=+335.201847850"
Sep 11 17:26:05.177338 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.177298    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:05.276707 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.276670    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:05.439993 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.439910    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"]
Sep 11 17:26:05.442332 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.442307    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.445259 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.445232    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 17:26:05.445388 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.445233    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-btsvq\""
Sep 11 17:26:05.457951 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.457927    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"]
Sep 11 17:26:05.487343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.487309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.487503 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.487355    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.487503 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.487438    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.487503 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.487462    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkv7q\" (UniqueName: \"kubernetes.io/projected/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-kube-api-access-vkv7q\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.487655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.487527    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.487655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.487556    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.487655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.487586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.487655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.487634    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.487850 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.487687    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.493276 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.493242    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:05.588727 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.588694    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.588905 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.588735    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.588905 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.588848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.588905 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.588892    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.589064 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.588913    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.589064 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.588944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.589169 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.589064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkv7q\" (UniqueName: \"kubernetes.io/projected/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-kube-api-access-vkv7q\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.589169 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.589137    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.589169 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.589159    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.589316 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.589170    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.589316 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.589197    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.589530 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.589510    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.589594 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.589549    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.589641 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.589602    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.591192 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.591175    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.591438 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.591423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.604925 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.604899    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkv7q\" (UniqueName: \"kubernetes.io/projected/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-kube-api-access-vkv7q\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.605620 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.605593    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc7czd2\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.611421 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.611395    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:05.752852 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.752771    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2" event={"ID":"c9c38b6d-7101-4132-ae6e-377a97881082","Type":"ContainerStarted","Data":"d9e2e3d66fec9e38abf899ab2a00de159298aed4b1a4634be4f781219af3ca7f"}
Sep 11 17:26:05.754404 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.754381    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:05.781329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:05.781280    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5897fcd6c7-qgbz2" podStartSLOduration=2.951791972 podStartE2EDuration="7.781262434s" totalImagesPullingTime="4.829470462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:26:05.780852286 +0000 UTC m=+336.213305088" watchObservedRunningTime="2026-09-11 17:26:05.781262434 +0000 UTC m=+336.213715253"
Sep 11 17:26:06.005979 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.005907    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"]
Sep 11 17:26:06.008656 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.008634    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.023803 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.023778    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"]
Sep 11 17:26:06.038020 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.037993    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"]
Sep 11 17:26:06.040176 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:26:06.040153    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1305c45a_0cac_40a2_9f51_f0b5a2aaa1b3.slice/crio-177e562bc2ab10a288320efdf4f889d91264de4c6ff8b6e6ce052c15c461dadf WatchSource:0}: Error finding container 177e562bc2ab10a288320efdf4f889d91264de4c6ff8b6e6ce052c15c461dadf: Status 404 returned error can't find the container with id 177e562bc2ab10a288320efdf4f889d91264de4c6ff8b6e6ce052c15c461dadf
Sep 11 17:26:06.042319 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.042279    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:06.094221 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.094188    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.094390 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.094291    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.094390 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.094326    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm678\" (UniqueName: \"kubernetes.io/projected/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-kube-api-access-hm678\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.094390 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.094368    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.094390 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.094385    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.094535 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.094408    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.094535 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.094466    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.094535 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.094507    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.094535 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.094532    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.195016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.194976    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm678\" (UniqueName: \"kubernetes.io/projected/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-kube-api-access-hm678\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.195016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.195018    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.195323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.195037    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.195323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.195059    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.195323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.195123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.195323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.195153    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.195323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.195177    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.195323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.195211    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.195323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.195259    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.195685 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.195469    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.195748 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.195691    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.195833 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.195814    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.195893 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.195858    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.196049 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.196023    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.197680 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.197659    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.197766 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.197680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.207329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.207309    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.207543 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.207525    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm678\" (UniqueName: \"kubernetes.io/projected/1f502bdb-2db6-47fb-a8a8-5b53d96519eb-kube-api-access-hm678\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg\" (UID: \"1f502bdb-2db6-47fb-a8a8-5b53d96519eb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.320085 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.319990    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:06.612112 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.612085    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"]
Sep 11 17:26:06.614367 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:26:06.614338    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f502bdb_2db6_47fb_a8a8_5b53d96519eb.slice/crio-888e5c2701f56da20b55bab9afebcd29410e793f64808737e87b651811e84c23 WatchSource:0}: Error finding container 888e5c2701f56da20b55bab9afebcd29410e793f64808737e87b651811e84c23: Status 404 returned error can't find the container with id 888e5c2701f56da20b55bab9afebcd29410e793f64808737e87b651811e84c23
Sep 11 17:26:06.763809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.763766    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2" event={"ID":"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3","Type":"ContainerStarted","Data":"177e562bc2ab10a288320efdf4f889d91264de4c6ff8b6e6ce052c15c461dadf"}
Sep 11 17:26:06.765268 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:06.765244    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg" event={"ID":"1f502bdb-2db6-47fb-a8a8-5b53d96519eb","Type":"ContainerStarted","Data":"888e5c2701f56da20b55bab9afebcd29410e793f64808737e87b651811e84c23"}
Sep 11 17:26:08.453874 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.453838    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"]
Sep 11 17:26:08.460257 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.460227    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:08.463558 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.463533    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 17:26:08.463690 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.463652    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 17:26:08.464985 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.464902    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 17:26:08.464985 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.464952    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-v94wv\""
Sep 11 17:26:08.465178 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.464957    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:26:08.465239 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.465207    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 17:26:08.467374 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.467352    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"]
Sep 11 17:26:08.519144 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.519114    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f5f8c43-f99c-41af-b9a0-8ebebf360ab6-metrics-cert\") pod \"lws-controller-manager-5786cf49b6-z49b8\" (UID: \"8f5f8c43-f99c-41af-b9a0-8ebebf360ab6\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:08.519279 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.519171    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8f5f8c43-f99c-41af-b9a0-8ebebf360ab6-manager-config\") pod \"lws-controller-manager-5786cf49b6-z49b8\" (UID: \"8f5f8c43-f99c-41af-b9a0-8ebebf360ab6\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:08.519349 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.519270    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncj6r\" (UniqueName: \"kubernetes.io/projected/8f5f8c43-f99c-41af-b9a0-8ebebf360ab6-kube-api-access-ncj6r\") pod \"lws-controller-manager-5786cf49b6-z49b8\" (UID: \"8f5f8c43-f99c-41af-b9a0-8ebebf360ab6\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:08.519349 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.519324    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f5f8c43-f99c-41af-b9a0-8ebebf360ab6-cert\") pod \"lws-controller-manager-5786cf49b6-z49b8\" (UID: \"8f5f8c43-f99c-41af-b9a0-8ebebf360ab6\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:08.619905 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.619872    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncj6r\" (UniqueName: \"kubernetes.io/projected/8f5f8c43-f99c-41af-b9a0-8ebebf360ab6-kube-api-access-ncj6r\") pod \"lws-controller-manager-5786cf49b6-z49b8\" (UID: \"8f5f8c43-f99c-41af-b9a0-8ebebf360ab6\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:08.619905 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.619909    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f5f8c43-f99c-41af-b9a0-8ebebf360ab6-cert\") pod \"lws-controller-manager-5786cf49b6-z49b8\" (UID: \"8f5f8c43-f99c-41af-b9a0-8ebebf360ab6\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:08.620112 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.619988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f5f8c43-f99c-41af-b9a0-8ebebf360ab6-metrics-cert\") pod \"lws-controller-manager-5786cf49b6-z49b8\" (UID: \"8f5f8c43-f99c-41af-b9a0-8ebebf360ab6\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:08.620112 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.620029    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8f5f8c43-f99c-41af-b9a0-8ebebf360ab6-manager-config\") pod \"lws-controller-manager-5786cf49b6-z49b8\" (UID: \"8f5f8c43-f99c-41af-b9a0-8ebebf360ab6\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:08.620883 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.620857    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8f5f8c43-f99c-41af-b9a0-8ebebf360ab6-manager-config\") pod \"lws-controller-manager-5786cf49b6-z49b8\" (UID: \"8f5f8c43-f99c-41af-b9a0-8ebebf360ab6\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:08.622700 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.622677    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f5f8c43-f99c-41af-b9a0-8ebebf360ab6-metrics-cert\") pod \"lws-controller-manager-5786cf49b6-z49b8\" (UID: \"8f5f8c43-f99c-41af-b9a0-8ebebf360ab6\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:08.622892 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.622871    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f5f8c43-f99c-41af-b9a0-8ebebf360ab6-cert\") pod \"lws-controller-manager-5786cf49b6-z49b8\" (UID: \"8f5f8c43-f99c-41af-b9a0-8ebebf360ab6\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:08.633066 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.633042    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncj6r\" (UniqueName: \"kubernetes.io/projected/8f5f8c43-f99c-41af-b9a0-8ebebf360ab6-kube-api-access-ncj6r\") pod \"lws-controller-manager-5786cf49b6-z49b8\" (UID: \"8f5f8c43-f99c-41af-b9a0-8ebebf360ab6\") " pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:08.773013 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:08.772924    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:09.372121 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:09.372067    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:26:09.372243 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:09.372165    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:26:09.372243 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:09.372201    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:26:09.378322 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:09.378291    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:26:09.378422 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:09.378371    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:26:09.378422 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:09.378399    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:26:09.560540 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:09.560507    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"]
Sep 11 17:26:09.562722 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:26:09.562692    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5f8c43_f99c_41af_b9a0_8ebebf360ab6.slice/crio-5b56f408f7d36875b9b4c209e6d0e125ea45eaddfbd1f6a74938db4004657289 WatchSource:0}: Error finding container 5b56f408f7d36875b9b4c209e6d0e125ea45eaddfbd1f6a74938db4004657289: Status 404 returned error can't find the container with id 5b56f408f7d36875b9b4c209e6d0e125ea45eaddfbd1f6a74938db4004657289
Sep 11 17:26:09.786547 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:09.786469    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2" event={"ID":"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3","Type":"ContainerStarted","Data":"a041a93cbd926ce9e417ede766fe42a93eced8d6b18bfe59117cca14871ffa60"}
Sep 11 17:26:09.788731 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:09.788696    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg" event={"ID":"1f502bdb-2db6-47fb-a8a8-5b53d96519eb","Type":"ContainerStarted","Data":"5b69610ece97d6a2af7e8662e2e55ed66985d661956889936f7ed4e6d0c7292c"}
Sep 11 17:26:09.790277 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:09.790243    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8" event={"ID":"8f5f8c43-f99c-41af-b9a0-8ebebf360ab6","Type":"ContainerStarted","Data":"5b56f408f7d36875b9b4c209e6d0e125ea45eaddfbd1f6a74938db4004657289"}
Sep 11 17:26:09.829689 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:09.829566    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2" podStartSLOduration=1.500407487 podStartE2EDuration="4.829530811s" totalImagesPullingTime="3.329123324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:26:09.824824202 +0000 UTC m=+340.257276981" watchObservedRunningTime="2026-09-11 17:26:09.829530811 +0000 UTC m=+340.261983587"
Sep 11 17:26:09.859872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:09.859817    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg" podStartSLOduration=2.098180473 podStartE2EDuration="4.859798537s" totalImagesPullingTime="2.761618064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:26:09.858805815 +0000 UTC m=+340.291258602" watchObservedRunningTime="2026-09-11 17:26:09.859798537 +0000 UTC m=+340.292251313"
Sep 11 17:26:10.320888 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:10.320851    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:10.391946 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:10.391910    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:10.593502 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:10.593110    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:10.702282 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:10.702237    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:10.755241 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:10.755203    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:10.756399 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:10.756375    2560 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 11 17:26:10.756565 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:10.756432    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2" podUID="1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 11 17:26:10.796509 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:10.796461    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:10.797444 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:10.797417    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg"
Sep 11 17:26:10.909878 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:10.909849    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"]
Sep 11 17:26:11.239141 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:11.239057    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:11.382606 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:11.382571    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:11.755622 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:11.755585    2560 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 11 17:26:11.756060 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:11.755641    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2" podUID="1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 11 17:26:11.917202 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:11.917165    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:12.018613 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:12.018554    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:12.213932 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:12.213897    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:12.755255 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:12.755210    2560 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 11 17:26:12.755445 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:12.755279    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2" podUID="1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 11 17:26:12.810110 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:12.810074    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8" event={"ID":"8f5f8c43-f99c-41af-b9a0-8ebebf360ab6","Type":"ContainerStarted","Data":"fed3b3e6a5258b99cce675b54acb6d5a315aac9739e889df960229109bbe5382"}
Sep 11 17:26:12.810639 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:12.810305    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:12.810639 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:12.810427    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2" podUID="1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" containerName="istio-proxy" containerID="cri-o://a041a93cbd926ce9e417ede766fe42a93eced8d6b18bfe59117cca14871ffa60" gracePeriod=30
Sep 11 17:26:12.830983 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:12.830929    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8" podStartSLOduration=2.391183921 podStartE2EDuration="4.830915618s" totalImagesPullingTime="2.439731697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:26:12.829304919 +0000 UTC m=+343.261757695" watchObservedRunningTime="2026-09-11 17:26:12.830915618 +0000 UTC m=+343.263368393"
Sep 11 17:26:15.755142 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:15.755116    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-d5vf9"
Sep 11 17:26:18.056559 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.056532    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:18.112832 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.112750    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-envoy\") pod \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") "
Sep 11 17:26:18.112832 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.112822    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-credential-socket\") pod \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") "
Sep 11 17:26:18.113049 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.112850    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-token\") pod \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") "
Sep 11 17:26:18.113049 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.112867    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkv7q\" (UniqueName: \"kubernetes.io/projected/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-kube-api-access-vkv7q\") pod \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") "
Sep 11 17:26:18.113049 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.113030    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-podinfo\") pod \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") "
Sep 11 17:26:18.113205 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.113094    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-data\") pod \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") "
Sep 11 17:26:18.113205 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.113119    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-workload-socket\") pod \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") "
Sep 11 17:26:18.113205 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.113145    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istiod-ca-cert\") pod \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") "
Sep 11 17:26:18.113205 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.113159    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" (UID: "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:26:18.113205 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.113194    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-workload-certs\") pod \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\" (UID: \"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3\") "
Sep 11 17:26:18.113453 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.113359    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" (UID: "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:26:18.113453 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.113372    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" (UID: "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:26:18.113678 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.113592    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-credential-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:26:18.113678 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.113615    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-data\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:26:18.113678 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.113630    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-workload-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:26:18.113678 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.113589    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" (UID: "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:26:18.113678 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.113658    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" (UID: "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:26:18.115209 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.115178    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" (UID: "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:26:18.115209 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.115198    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-kube-api-access-vkv7q" (OuterVolumeSpecName: "kube-api-access-vkv7q") pod "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" (UID: "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3"). InnerVolumeSpecName "kube-api-access-vkv7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:26:18.115361 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.115327    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" (UID: "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:26:18.115415 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.115399    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" (UID: "1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:26:18.214931 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.214893    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-podinfo\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:26:18.214931 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.214924    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istiod-ca-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:26:18.214931 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.214936    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-workload-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:26:18.215169 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.214949    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-envoy\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:26:18.215169 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.214959    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-istio-token\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:26:18.215169 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.214970    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkv7q\" (UniqueName: \"kubernetes.io/projected/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3-kube-api-access-vkv7q\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:26:18.849278 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.849246    2560 generic.go:356] "Generic (PLEG): container finished" podID="1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" containerID="a041a93cbd926ce9e417ede766fe42a93eced8d6b18bfe59117cca14871ffa60" exitCode=0
Sep 11 17:26:18.849496 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.849304    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"
Sep 11 17:26:18.849496 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.849332    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2" event={"ID":"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3","Type":"ContainerDied","Data":"a041a93cbd926ce9e417ede766fe42a93eced8d6b18bfe59117cca14871ffa60"}
Sep 11 17:26:18.849496 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.849388    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2" event={"ID":"1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3","Type":"ContainerDied","Data":"177e562bc2ab10a288320efdf4f889d91264de4c6ff8b6e6ce052c15c461dadf"}
Sep 11 17:26:18.849496 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.849409    2560 scope.go:117] "RemoveContainer" containerID="a041a93cbd926ce9e417ede766fe42a93eced8d6b18bfe59117cca14871ffa60"
Sep 11 17:26:18.858874 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.858854    2560 scope.go:117] "RemoveContainer" containerID="a041a93cbd926ce9e417ede766fe42a93eced8d6b18bfe59117cca14871ffa60"
Sep 11 17:26:18.859162 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:26:18.859145    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a041a93cbd926ce9e417ede766fe42a93eced8d6b18bfe59117cca14871ffa60\": container with ID starting with a041a93cbd926ce9e417ede766fe42a93eced8d6b18bfe59117cca14871ffa60 not found: ID does not exist" containerID="a041a93cbd926ce9e417ede766fe42a93eced8d6b18bfe59117cca14871ffa60"
Sep 11 17:26:18.859205 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.859171    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a041a93cbd926ce9e417ede766fe42a93eced8d6b18bfe59117cca14871ffa60"} err="failed to get container status \"a041a93cbd926ce9e417ede766fe42a93eced8d6b18bfe59117cca14871ffa60\": rpc error: code = NotFound desc = could not find container \"a041a93cbd926ce9e417ede766fe42a93eced8d6b18bfe59117cca14871ffa60\": container with ID starting with a041a93cbd926ce9e417ede766fe42a93eced8d6b18bfe59117cca14871ffa60 not found: ID does not exist"
Sep 11 17:26:18.877995 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.877966    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"]
Sep 11 17:26:18.883398 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:18.883374    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc7czd2"]
Sep 11 17:26:20.050808 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:20.050776    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" path="/var/lib/kubelet/pods/1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3/volumes"
Sep 11 17:26:21.718917 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:21.718885    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:21.895078 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:21.895027    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:23.817938 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:23.817905    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5786cf49b6-z49b8"
Sep 11 17:26:27.342571 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:27.342540    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:27.415917 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:27.415881    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:29.399965 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:29.399933    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:29.471403 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:29.471376    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:26:59.960766 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:26:59.960723    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:00.058572 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:00.058539    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:11.458051 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:11.458023    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:11.554387 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:11.554357    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:15.027623 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:15.027584    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:17.792347 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:17.792310    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:19.608400 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:19.608370    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:19.993905 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:19.993876    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:20.389885 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:20.389849    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:20.643998 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:20.643919    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:20.896118 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:20.896040    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:21.648428 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:21.648400    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:21.731751 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:21.731722    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:22.652034 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:22.652001    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b5d5b8444-7v7vt"]
Sep 11 17:27:22.770659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:22.770633    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:22.872383 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:22.872351    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:22.985556 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:22.985468    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:23.085529 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:23.085498    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:23.816210 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:23.816178    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:24.186927 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:24.186892    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:25.645717 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:25.645680    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:25.767675 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:25.767641    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:27.611741 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:27.611714    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:27.713586 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:27.713554    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:31.115879 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:31.115842    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:31.715827 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:31.715795    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:31.816844 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:31.816812    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:35.255266 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:35.255239    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:35.412917 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:35.412884    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:35.488540 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:35.488457    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:44.068343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:44.068309    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:44.174411 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:44.174381    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:46.623135 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:46.623107    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:46.722321 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:46.722289    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:47.671519 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:47.671456    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b5d5b8444-7v7vt" podUID="020a3b4b-e11f-4cf4-9178-9b787de8ddcd" containerName="console" containerID="cri-o://a1d2e02df00b98482c2350070ec8edd60475a27130fee9cdc4c940980a0a66c3" gracePeriod=15
Sep 11 17:27:47.919348 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:47.919326    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:27:48.002420 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.002341    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-trusted-ca-bundle\") pod \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") "
Sep 11 17:27:48.002420 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.002384    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-service-ca\") pod \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") "
Sep 11 17:27:48.002420 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.002404    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-serving-cert\") pod \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") "
Sep 11 17:27:48.002716 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.002452    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tpx7\" (UniqueName: \"kubernetes.io/projected/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-kube-api-access-7tpx7\") pod \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") "
Sep 11 17:27:48.002716 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.002501    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-oauth-config\") pod \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") "
Sep 11 17:27:48.002716 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.002523    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-oauth-serving-cert\") pod \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") "
Sep 11 17:27:48.002716 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.002549    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-config\") pod \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\" (UID: \"020a3b4b-e11f-4cf4-9178-9b787de8ddcd\") "
Sep 11 17:27:48.002916 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.002803    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "020a3b4b-e11f-4cf4-9178-9b787de8ddcd" (UID: "020a3b4b-e11f-4cf4-9178-9b787de8ddcd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:27:48.002967 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.002941    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-service-ca" (OuterVolumeSpecName: "service-ca") pod "020a3b4b-e11f-4cf4-9178-9b787de8ddcd" (UID: "020a3b4b-e11f-4cf4-9178-9b787de8ddcd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:27:48.003032 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.002997    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-config" (OuterVolumeSpecName: "console-config") pod "020a3b4b-e11f-4cf4-9178-9b787de8ddcd" (UID: "020a3b4b-e11f-4cf4-9178-9b787de8ddcd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:27:48.003133 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.003106    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "020a3b4b-e11f-4cf4-9178-9b787de8ddcd" (UID: "020a3b4b-e11f-4cf4-9178-9b787de8ddcd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:27:48.004751 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.004720    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "020a3b4b-e11f-4cf4-9178-9b787de8ddcd" (UID: "020a3b4b-e11f-4cf4-9178-9b787de8ddcd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:27:48.004860 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.004787    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "020a3b4b-e11f-4cf4-9178-9b787de8ddcd" (UID: "020a3b4b-e11f-4cf4-9178-9b787de8ddcd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:27:48.004860 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.004818    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-kube-api-access-7tpx7" (OuterVolumeSpecName: "kube-api-access-7tpx7") pod "020a3b4b-e11f-4cf4-9178-9b787de8ddcd" (UID: "020a3b4b-e11f-4cf4-9178-9b787de8ddcd"). InnerVolumeSpecName "kube-api-access-7tpx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:27:48.104151 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.104112    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tpx7\" (UniqueName: \"kubernetes.io/projected/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-kube-api-access-7tpx7\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:27:48.104151 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.104145    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-oauth-config\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:27:48.104151 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.104160    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-oauth-serving-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:27:48.104446 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.104174    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-config\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:27:48.104446 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.104188    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-trusted-ca-bundle\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:27:48.104446 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.104199    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:27:48.104446 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.104211    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/020a3b4b-e11f-4cf4-9178-9b787de8ddcd-console-serving-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:27:48.245221 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.245186    2560 generic.go:356] "Generic (PLEG): container finished" podID="020a3b4b-e11f-4cf4-9178-9b787de8ddcd" containerID="a1d2e02df00b98482c2350070ec8edd60475a27130fee9cdc4c940980a0a66c3" exitCode=0
Sep 11 17:27:48.245221 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.245225    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b5d5b8444-7v7vt"
Sep 11 17:27:48.245437 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.245274    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5d5b8444-7v7vt" event={"ID":"020a3b4b-e11f-4cf4-9178-9b787de8ddcd","Type":"ContainerDied","Data":"a1d2e02df00b98482c2350070ec8edd60475a27130fee9cdc4c940980a0a66c3"}
Sep 11 17:27:48.245437 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.245321    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b5d5b8444-7v7vt" event={"ID":"020a3b4b-e11f-4cf4-9178-9b787de8ddcd","Type":"ContainerDied","Data":"70f1e1db815a3e19d565000b0ae18bbe5ba98f75cd2b90c9ae47a58fbfcee498"}
Sep 11 17:27:48.245437 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.245343    2560 scope.go:117] "RemoveContainer" containerID="a1d2e02df00b98482c2350070ec8edd60475a27130fee9cdc4c940980a0a66c3"
Sep 11 17:27:48.254072 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.254060    2560 scope.go:117] "RemoveContainer" containerID="a1d2e02df00b98482c2350070ec8edd60475a27130fee9cdc4c940980a0a66c3"
Sep 11 17:27:48.254369 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:27:48.254351    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d2e02df00b98482c2350070ec8edd60475a27130fee9cdc4c940980a0a66c3\": container with ID starting with a1d2e02df00b98482c2350070ec8edd60475a27130fee9cdc4c940980a0a66c3 not found: ID does not exist" containerID="a1d2e02df00b98482c2350070ec8edd60475a27130fee9cdc4c940980a0a66c3"
Sep 11 17:27:48.254417 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.254378    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d2e02df00b98482c2350070ec8edd60475a27130fee9cdc4c940980a0a66c3"} err="failed to get container status \"a1d2e02df00b98482c2350070ec8edd60475a27130fee9cdc4c940980a0a66c3\": rpc error: code = NotFound desc = could not find container \"a1d2e02df00b98482c2350070ec8edd60475a27130fee9cdc4c940980a0a66c3\": container with ID starting with a1d2e02df00b98482c2350070ec8edd60475a27130fee9cdc4c940980a0a66c3 not found: ID does not exist"
Sep 11 17:27:48.266488 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.266456    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b5d5b8444-7v7vt"]
Sep 11 17:27:48.273290 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:48.271279    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b5d5b8444-7v7vt"]
Sep 11 17:27:49.824146 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:49.824114    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:49.923757 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:49.923716    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:50.050398 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:50.050371    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="020a3b4b-e11f-4cf4-9178-9b787de8ddcd" path="/var/lib/kubelet/pods/020a3b4b-e11f-4cf4-9178-9b787de8ddcd/volumes"
Sep 11 17:27:51.937416 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:51.937385    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:52.037384 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:52.037348    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:54.225874 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:54.225833    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:54.326552 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:54.326467    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:54.699391 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:54.699360    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:54.771959 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:54.771923    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:55.434285 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:55.434250    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:27:55.553347 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:27:55.553314    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:03.028691 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:03.028657    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:03.128741 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:03.128706    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:14.095957 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.095908    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"]
Sep 11 17:28:14.096430 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.096396    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020a3b4b-e11f-4cf4-9178-9b787de8ddcd" containerName="console"
Sep 11 17:28:14.096430 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.096413    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="020a3b4b-e11f-4cf4-9178-9b787de8ddcd" containerName="console"
Sep 11 17:28:14.096568 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.096450    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" containerName="istio-proxy"
Sep 11 17:28:14.096568 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.096458    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" containerName="istio-proxy"
Sep 11 17:28:14.096635 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.096585    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="020a3b4b-e11f-4cf4-9178-9b787de8ddcd" containerName="console"
Sep 11 17:28:14.096635 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.096601    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1305c45a-0cac-40a2-9f51-f0b5a2aaa1b3" containerName="istio-proxy"
Sep 11 17:28:14.099830 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.099811    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.102519 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.102496    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hzskt\""
Sep 11 17:28:14.111638 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.111616    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"]
Sep 11 17:28:14.138520 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.138464    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:14.236363 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.236321    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8555f-802e-4533-8e21-d1eb611ba4a6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.236363 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.236369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d8555f-802e-4533-8e21-d1eb611ba4a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.236577 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.236390    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d8555f-802e-4533-8e21-d1eb611ba4a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.236577 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.236462    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8555f-802e-4533-8e21-d1eb611ba4a6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.236577 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.236533    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d8555f-802e-4533-8e21-d1eb611ba4a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.236577 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.236570    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24d8555f-802e-4533-8e21-d1eb611ba4a6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.236699 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.236589    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d8555f-802e-4533-8e21-d1eb611ba4a6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.236699 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.236648    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d8555f-802e-4533-8e21-d1eb611ba4a6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.236699 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.236673    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq6kw\" (UniqueName: \"kubernetes.io/projected/24d8555f-802e-4533-8e21-d1eb611ba4a6-kube-api-access-nq6kw\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.338088 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.338050    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8555f-802e-4533-8e21-d1eb611ba4a6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.338088 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.338090    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d8555f-802e-4533-8e21-d1eb611ba4a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.338313 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.338106    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d8555f-802e-4533-8e21-d1eb611ba4a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.338313 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.338126    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8555f-802e-4533-8e21-d1eb611ba4a6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.338313 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.338148    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d8555f-802e-4533-8e21-d1eb611ba4a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.338313 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.338175    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24d8555f-802e-4533-8e21-d1eb611ba4a6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.338313 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.338195    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d8555f-802e-4533-8e21-d1eb611ba4a6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.338313 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.338224    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d8555f-802e-4533-8e21-d1eb611ba4a6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.338313 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.338262    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq6kw\" (UniqueName: \"kubernetes.io/projected/24d8555f-802e-4533-8e21-d1eb611ba4a6-kube-api-access-nq6kw\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.338709 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.338556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8555f-802e-4533-8e21-d1eb611ba4a6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.338838 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.338780    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24d8555f-802e-4533-8e21-d1eb611ba4a6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.339104 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.338960    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24d8555f-802e-4533-8e21-d1eb611ba4a6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.339104 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.338994    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24d8555f-802e-4533-8e21-d1eb611ba4a6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.339104 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.338995    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24d8555f-802e-4533-8e21-d1eb611ba4a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.341069 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.341042    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24d8555f-802e-4533-8e21-d1eb611ba4a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.341187 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.341169    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24d8555f-802e-4533-8e21-d1eb611ba4a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.346696 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.346641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24d8555f-802e-4533-8e21-d1eb611ba4a6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.346856 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.346837    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq6kw\" (UniqueName: \"kubernetes.io/projected/24d8555f-802e-4533-8e21-d1eb611ba4a6-kube-api-access-nq6kw\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wmvtc\" (UID: \"24d8555f-802e-4533-8e21-d1eb611ba4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.410260 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.410221    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:14.480091 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.480065    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:14.686108 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.686073    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"]
Sep 11 17:28:14.689457 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:28:14.689430    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d8555f_802e_4533_8e21_d1eb611ba4a6.slice/crio-0678710dd36ace564878d458a8c9afd8f346b7d525cb95dfdb5cb99fa4477161 WatchSource:0}: Error finding container 0678710dd36ace564878d458a8c9afd8f346b7d525cb95dfdb5cb99fa4477161: Status 404 returned error can't find the container with id 0678710dd36ace564878d458a8c9afd8f346b7d525cb95dfdb5cb99fa4477161
Sep 11 17:28:14.691643 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.691611    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:28:14.691724 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.691680    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:28:14.691724 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:14.691707    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:28:15.365069 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:15.365036    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc" event={"ID":"24d8555f-802e-4533-8e21-d1eb611ba4a6","Type":"ContainerStarted","Data":"6d773a838218ec7dc6f9b15b3fe26547b3808e4a0e0c6a3803d31338b25a74ab"}
Sep 11 17:28:15.365576 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:15.365088    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc" event={"ID":"24d8555f-802e-4533-8e21-d1eb611ba4a6","Type":"ContainerStarted","Data":"0678710dd36ace564878d458a8c9afd8f346b7d525cb95dfdb5cb99fa4477161"}
Sep 11 17:28:15.389894 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:15.389838    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc" podStartSLOduration=1.389822564 podStartE2EDuration="1.389822564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:28:15.388642999 +0000 UTC m=+465.821095777" watchObservedRunningTime="2026-09-11 17:28:15.389822564 +0000 UTC m=+465.822275403"
Sep 11 17:28:15.410419 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:15.410385    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:15.415168 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:15.415145    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:16.370073 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:16.370043    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:16.371075 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:16.371059    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:28:28.082120 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:28.082086    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:28.183646 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:28.183614    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:28.239286 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:28.239259    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:28.853195 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:28.853164    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-r8tgz"]
Sep 11 17:28:28.856351 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:28.856329    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-r8tgz"
Sep 11 17:28:28.858786 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:28.858760    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 17:28:28.858786 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:28.858770    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 17:28:28.859725 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:28.859711    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bcm5t\""
Sep 11 17:28:28.862728 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:28.862708    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-r8tgz"]
Sep 11 17:28:28.916035 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:28.916008    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:28.967324 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:28.967295    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m954q\" (UniqueName: \"kubernetes.io/projected/29bed29b-c4d4-4184-83dc-df315acaf267-kube-api-access-m954q\") pod \"authorino-7bcfd8f9cd-r8tgz\" (UID: \"29bed29b-c4d4-4184-83dc-df315acaf267\") " pod="kuadrant-system/authorino-7bcfd8f9cd-r8tgz"
Sep 11 17:28:28.971835 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:28.971810    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-22nxx"]
Sep 11 17:28:28.975132 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:28.975112    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-22nxx"
Sep 11 17:28:28.982301 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:28.982277    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-22nxx"]
Sep 11 17:28:28.985799 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:28.985778    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:29.068005 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:29.067983    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m954q\" (UniqueName: \"kubernetes.io/projected/29bed29b-c4d4-4184-83dc-df315acaf267-kube-api-access-m954q\") pod \"authorino-7bcfd8f9cd-r8tgz\" (UID: \"29bed29b-c4d4-4184-83dc-df315acaf267\") " pod="kuadrant-system/authorino-7bcfd8f9cd-r8tgz"
Sep 11 17:28:29.068147 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:29.068069    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sp5z\" (UniqueName: \"kubernetes.io/projected/b8215939-16f4-42f1-ad5f-c9adbfb3d532-kube-api-access-4sp5z\") pod \"authorino-59c974bd48-22nxx\" (UID: \"b8215939-16f4-42f1-ad5f-c9adbfb3d532\") " pod="kuadrant-system/authorino-59c974bd48-22nxx"
Sep 11 17:28:29.076867 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:29.076843    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m954q\" (UniqueName: \"kubernetes.io/projected/29bed29b-c4d4-4184-83dc-df315acaf267-kube-api-access-m954q\") pod \"authorino-7bcfd8f9cd-r8tgz\" (UID: \"29bed29b-c4d4-4184-83dc-df315acaf267\") " pod="kuadrant-system/authorino-7bcfd8f9cd-r8tgz"
Sep 11 17:28:29.166419 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:29.166395    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-r8tgz"
Sep 11 17:28:29.168804 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:29.168782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sp5z\" (UniqueName: \"kubernetes.io/projected/b8215939-16f4-42f1-ad5f-c9adbfb3d532-kube-api-access-4sp5z\") pod \"authorino-59c974bd48-22nxx\" (UID: \"b8215939-16f4-42f1-ad5f-c9adbfb3d532\") " pod="kuadrant-system/authorino-59c974bd48-22nxx"
Sep 11 17:28:29.178022 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:29.178000    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sp5z\" (UniqueName: \"kubernetes.io/projected/b8215939-16f4-42f1-ad5f-c9adbfb3d532-kube-api-access-4sp5z\") pod \"authorino-59c974bd48-22nxx\" (UID: \"b8215939-16f4-42f1-ad5f-c9adbfb3d532\") " pod="kuadrant-system/authorino-59c974bd48-22nxx"
Sep 11 17:28:29.283848 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:29.283819    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-22nxx"
Sep 11 17:28:29.435643 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:29.435619    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-r8tgz"]
Sep 11 17:28:29.437505 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:28:29.437466    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29bed29b_c4d4_4184_83dc_df315acaf267.slice/crio-72a634affa88ea48628806ff87dfeb9c82cc609f7294c5d647c46f267ebd885c WatchSource:0}: Error finding container 72a634affa88ea48628806ff87dfeb9c82cc609f7294c5d647c46f267ebd885c: Status 404 returned error can't find the container with id 72a634affa88ea48628806ff87dfeb9c82cc609f7294c5d647c46f267ebd885c
Sep 11 17:28:29.549182 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:29.549162    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-22nxx"]
Sep 11 17:28:29.550853 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:28:29.550818    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8215939_16f4_42f1_ad5f_c9adbfb3d532.slice/crio-1d2a68d572bace5af09bb80af92019f7d6bab3726bd4ccd48a687ee15d536408 WatchSource:0}: Error finding container 1d2a68d572bace5af09bb80af92019f7d6bab3726bd4ccd48a687ee15d536408: Status 404 returned error can't find the container with id 1d2a68d572bace5af09bb80af92019f7d6bab3726bd4ccd48a687ee15d536408
Sep 11 17:28:30.435204 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:30.435120    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-r8tgz" event={"ID":"29bed29b-c4d4-4184-83dc-df315acaf267","Type":"ContainerStarted","Data":"72a634affa88ea48628806ff87dfeb9c82cc609f7294c5d647c46f267ebd885c"}
Sep 11 17:28:30.437830 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:30.437729    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-22nxx" event={"ID":"b8215939-16f4-42f1-ad5f-c9adbfb3d532","Type":"ContainerStarted","Data":"1d2a68d572bace5af09bb80af92019f7d6bab3726bd4ccd48a687ee15d536408"}
Sep 11 17:28:32.448868 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:32.448832    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-r8tgz" event={"ID":"29bed29b-c4d4-4184-83dc-df315acaf267","Type":"ContainerStarted","Data":"d9743c3a9317fdfd8c0e29d860dca4619749edbd92dfc4314db2bace37d3e751"}
Sep 11 17:28:32.450239 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:32.450215    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-22nxx" event={"ID":"b8215939-16f4-42f1-ad5f-c9adbfb3d532","Type":"ContainerStarted","Data":"a9f0f8abec37c399c96ddeb86bca3a87272f4e5dc2606500a58b627e7659d255"}
Sep 11 17:28:32.464367 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:32.464324    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-r8tgz" podStartSLOduration=1.88862445 podStartE2EDuration="4.464310126s" totalImagesPullingTime="2.575685676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:28:32.463509741 +0000 UTC m=+482.895962519" watchObservedRunningTime="2026-09-11 17:28:32.464310126 +0000 UTC m=+482.896762902"
Sep 11 17:28:32.476843 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:32.476792    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-22nxx" podStartSLOduration=2.024272581 podStartE2EDuration="4.476772723s" totalImagesPullingTime="2.452500142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:28:32.476361201 +0000 UTC m=+482.908813980" watchObservedRunningTime="2026-09-11 17:28:32.476772723 +0000 UTC m=+482.909225501"
Sep 11 17:28:32.503567 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:32.503543    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-r8tgz"]
Sep 11 17:28:34.459602 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:34.459549    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-r8tgz" podUID="29bed29b-c4d4-4184-83dc-df315acaf267" containerName="authorino" containerID="cri-o://d9743c3a9317fdfd8c0e29d860dca4619749edbd92dfc4314db2bace37d3e751" gracePeriod=30
Sep 11 17:28:34.703746 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:34.703724    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-r8tgz"
Sep 11 17:28:34.821766 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:34.821689    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m954q\" (UniqueName: \"kubernetes.io/projected/29bed29b-c4d4-4184-83dc-df315acaf267-kube-api-access-m954q\") pod \"29bed29b-c4d4-4184-83dc-df315acaf267\" (UID: \"29bed29b-c4d4-4184-83dc-df315acaf267\") "
Sep 11 17:28:34.823847 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:34.823822    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bed29b-c4d4-4184-83dc-df315acaf267-kube-api-access-m954q" (OuterVolumeSpecName: "kube-api-access-m954q") pod "29bed29b-c4d4-4184-83dc-df315acaf267" (UID: "29bed29b-c4d4-4184-83dc-df315acaf267"). InnerVolumeSpecName "kube-api-access-m954q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:28:34.922538 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:34.922502    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m954q\" (UniqueName: \"kubernetes.io/projected/29bed29b-c4d4-4184-83dc-df315acaf267-kube-api-access-m954q\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:28:35.465623 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:35.465580    2560 generic.go:356] "Generic (PLEG): container finished" podID="29bed29b-c4d4-4184-83dc-df315acaf267" containerID="d9743c3a9317fdfd8c0e29d860dca4619749edbd92dfc4314db2bace37d3e751" exitCode=0
Sep 11 17:28:35.466039 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:35.465635    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-r8tgz"
Sep 11 17:28:35.466039 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:35.465634    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-r8tgz" event={"ID":"29bed29b-c4d4-4184-83dc-df315acaf267","Type":"ContainerDied","Data":"d9743c3a9317fdfd8c0e29d860dca4619749edbd92dfc4314db2bace37d3e751"}
Sep 11 17:28:35.466039 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:35.465752    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-r8tgz" event={"ID":"29bed29b-c4d4-4184-83dc-df315acaf267","Type":"ContainerDied","Data":"72a634affa88ea48628806ff87dfeb9c82cc609f7294c5d647c46f267ebd885c"}
Sep 11 17:28:35.466039 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:35.465775    2560 scope.go:117] "RemoveContainer" containerID="d9743c3a9317fdfd8c0e29d860dca4619749edbd92dfc4314db2bace37d3e751"
Sep 11 17:28:35.473679 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:35.473661    2560 scope.go:117] "RemoveContainer" containerID="d9743c3a9317fdfd8c0e29d860dca4619749edbd92dfc4314db2bace37d3e751"
Sep 11 17:28:35.473948 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:28:35.473930    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9743c3a9317fdfd8c0e29d860dca4619749edbd92dfc4314db2bace37d3e751\": container with ID starting with d9743c3a9317fdfd8c0e29d860dca4619749edbd92dfc4314db2bace37d3e751 not found: ID does not exist" containerID="d9743c3a9317fdfd8c0e29d860dca4619749edbd92dfc4314db2bace37d3e751"
Sep 11 17:28:35.473991 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:35.473956    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9743c3a9317fdfd8c0e29d860dca4619749edbd92dfc4314db2bace37d3e751"} err="failed to get container status \"d9743c3a9317fdfd8c0e29d860dca4619749edbd92dfc4314db2bace37d3e751\": rpc error: code = NotFound desc = could not find container \"d9743c3a9317fdfd8c0e29d860dca4619749edbd92dfc4314db2bace37d3e751\": container with ID starting with d9743c3a9317fdfd8c0e29d860dca4619749edbd92dfc4314db2bace37d3e751 not found: ID does not exist"
Sep 11 17:28:35.485995 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:35.485971    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-r8tgz"]
Sep 11 17:28:35.489518 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:35.489498    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-r8tgz"]
Sep 11 17:28:36.050518 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:36.050467    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29bed29b-c4d4-4184-83dc-df315acaf267" path="/var/lib/kubelet/pods/29bed29b-c4d4-4184-83dc-df315acaf267/volumes"
Sep 11 17:28:47.801541 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:47.801497    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:47.903917 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:47.903874    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:28:48.241680 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:28:48.241642    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:00.367125 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.367092    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:00.516813 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.516757    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:00.669108 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.669073    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-n2x27"]
Sep 11 17:29:00.669508 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.669489    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29bed29b-c4d4-4184-83dc-df315acaf267" containerName="authorino"
Sep 11 17:29:00.669600 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.669510    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="29bed29b-c4d4-4184-83dc-df315acaf267" containerName="authorino"
Sep 11 17:29:00.669652 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.669608    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="29bed29b-c4d4-4184-83dc-df315acaf267" containerName="authorino"
Sep 11 17:29:00.672396 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.672376    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-n2x27"
Sep 11 17:29:00.674720 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.674698    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 17:29:00.679930 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.679905    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-n2x27"]
Sep 11 17:29:00.762461 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.762424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns55v\" (UniqueName: \"kubernetes.io/projected/11e1b64a-9779-4472-a24d-59522ebc5a0e-kube-api-access-ns55v\") pod \"authorino-6d4dfc649f-n2x27\" (UID: \"11e1b64a-9779-4472-a24d-59522ebc5a0e\") " pod="kuadrant-system/authorino-6d4dfc649f-n2x27"
Sep 11 17:29:00.762680 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.762518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/11e1b64a-9779-4472-a24d-59522ebc5a0e-tls-cert\") pod \"authorino-6d4dfc649f-n2x27\" (UID: \"11e1b64a-9779-4472-a24d-59522ebc5a0e\") " pod="kuadrant-system/authorino-6d4dfc649f-n2x27"
Sep 11 17:29:00.816231 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.816196    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-n2x27"]
Sep 11 17:29:00.816511 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:29:00.816491    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ns55v tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-n2x27" podUID="11e1b64a-9779-4472-a24d-59522ebc5a0e"
Sep 11 17:29:00.839498 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.839461    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-88fb69696-qptsg"]
Sep 11 17:29:00.842799 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.842778    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-88fb69696-qptsg"
Sep 11 17:29:00.852113 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.852081    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-88fb69696-qptsg"]
Sep 11 17:29:00.863763 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.863729    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvqhn\" (UniqueName: \"kubernetes.io/projected/42028f3d-7343-45af-a0c7-15c59bf4a40c-kube-api-access-xvqhn\") pod \"authorino-88fb69696-qptsg\" (UID: \"42028f3d-7343-45af-a0c7-15c59bf4a40c\") " pod="kuadrant-system/authorino-88fb69696-qptsg"
Sep 11 17:29:00.863914 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.863778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/11e1b64a-9779-4472-a24d-59522ebc5a0e-tls-cert\") pod \"authorino-6d4dfc649f-n2x27\" (UID: \"11e1b64a-9779-4472-a24d-59522ebc5a0e\") " pod="kuadrant-system/authorino-6d4dfc649f-n2x27"
Sep 11 17:29:00.863914 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.863803    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/42028f3d-7343-45af-a0c7-15c59bf4a40c-tls-cert\") pod \"authorino-88fb69696-qptsg\" (UID: \"42028f3d-7343-45af-a0c7-15c59bf4a40c\") " pod="kuadrant-system/authorino-88fb69696-qptsg"
Sep 11 17:29:00.863914 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.863883    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns55v\" (UniqueName: \"kubernetes.io/projected/11e1b64a-9779-4472-a24d-59522ebc5a0e-kube-api-access-ns55v\") pod \"authorino-6d4dfc649f-n2x27\" (UID: \"11e1b64a-9779-4472-a24d-59522ebc5a0e\") " pod="kuadrant-system/authorino-6d4dfc649f-n2x27"
Sep 11 17:29:00.866313 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.866286    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/11e1b64a-9779-4472-a24d-59522ebc5a0e-tls-cert\") pod \"authorino-6d4dfc649f-n2x27\" (UID: \"11e1b64a-9779-4472-a24d-59522ebc5a0e\") " pod="kuadrant-system/authorino-6d4dfc649f-n2x27"
Sep 11 17:29:00.871830 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.871796    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns55v\" (UniqueName: \"kubernetes.io/projected/11e1b64a-9779-4472-a24d-59522ebc5a0e-kube-api-access-ns55v\") pod \"authorino-6d4dfc649f-n2x27\" (UID: \"11e1b64a-9779-4472-a24d-59522ebc5a0e\") " pod="kuadrant-system/authorino-6d4dfc649f-n2x27"
Sep 11 17:29:00.964667 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.964576    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvqhn\" (UniqueName: \"kubernetes.io/projected/42028f3d-7343-45af-a0c7-15c59bf4a40c-kube-api-access-xvqhn\") pod \"authorino-88fb69696-qptsg\" (UID: \"42028f3d-7343-45af-a0c7-15c59bf4a40c\") " pod="kuadrant-system/authorino-88fb69696-qptsg"
Sep 11 17:29:00.964667 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.964623    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/42028f3d-7343-45af-a0c7-15c59bf4a40c-tls-cert\") pod \"authorino-88fb69696-qptsg\" (UID: \"42028f3d-7343-45af-a0c7-15c59bf4a40c\") " pod="kuadrant-system/authorino-88fb69696-qptsg"
Sep 11 17:29:00.967066 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.967037    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/42028f3d-7343-45af-a0c7-15c59bf4a40c-tls-cert\") pod \"authorino-88fb69696-qptsg\" (UID: \"42028f3d-7343-45af-a0c7-15c59bf4a40c\") " pod="kuadrant-system/authorino-88fb69696-qptsg"
Sep 11 17:29:00.972829 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:00.972806    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvqhn\" (UniqueName: \"kubernetes.io/projected/42028f3d-7343-45af-a0c7-15c59bf4a40c-kube-api-access-xvqhn\") pod \"authorino-88fb69696-qptsg\" (UID: \"42028f3d-7343-45af-a0c7-15c59bf4a40c\") " pod="kuadrant-system/authorino-88fb69696-qptsg"
Sep 11 17:29:01.158240 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:01.158206    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-88fb69696-qptsg"
Sep 11 17:29:01.425786 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:01.425751    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-88fb69696-qptsg"]
Sep 11 17:29:01.428536 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:29:01.428508    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42028f3d_7343_45af_a0c7_15c59bf4a40c.slice/crio-5f5165c50ef02e799af6030db2f832c219c95c4b6769a3b586862390a6ee6d5c WatchSource:0}: Error finding container 5f5165c50ef02e799af6030db2f832c219c95c4b6769a3b586862390a6ee6d5c: Status 404 returned error can't find the container with id 5f5165c50ef02e799af6030db2f832c219c95c4b6769a3b586862390a6ee6d5c
Sep 11 17:29:01.578229 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:01.578204    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-n2x27"
Sep 11 17:29:01.578374 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:01.578196    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-88fb69696-qptsg" event={"ID":"42028f3d-7343-45af-a0c7-15c59bf4a40c","Type":"ContainerStarted","Data":"5f5165c50ef02e799af6030db2f832c219c95c4b6769a3b586862390a6ee6d5c"}
Sep 11 17:29:01.586233 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:01.586213    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-n2x27"
Sep 11 17:29:01.669866 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:01.669840    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/11e1b64a-9779-4472-a24d-59522ebc5a0e-tls-cert\") pod \"11e1b64a-9779-4472-a24d-59522ebc5a0e\" (UID: \"11e1b64a-9779-4472-a24d-59522ebc5a0e\") "
Sep 11 17:29:01.669999 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:01.669910    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns55v\" (UniqueName: \"kubernetes.io/projected/11e1b64a-9779-4472-a24d-59522ebc5a0e-kube-api-access-ns55v\") pod \"11e1b64a-9779-4472-a24d-59522ebc5a0e\" (UID: \"11e1b64a-9779-4472-a24d-59522ebc5a0e\") "
Sep 11 17:29:01.671845 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:01.671816    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11e1b64a-9779-4472-a24d-59522ebc5a0e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "11e1b64a-9779-4472-a24d-59522ebc5a0e" (UID: "11e1b64a-9779-4472-a24d-59522ebc5a0e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:29:01.671937 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:01.671874    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e1b64a-9779-4472-a24d-59522ebc5a0e-kube-api-access-ns55v" (OuterVolumeSpecName: "kube-api-access-ns55v") pod "11e1b64a-9779-4472-a24d-59522ebc5a0e" (UID: "11e1b64a-9779-4472-a24d-59522ebc5a0e"). InnerVolumeSpecName "kube-api-access-ns55v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:29:01.771368 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:01.771297    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns55v\" (UniqueName: \"kubernetes.io/projected/11e1b64a-9779-4472-a24d-59522ebc5a0e-kube-api-access-ns55v\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:29:01.771368 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:01.771324    2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/11e1b64a-9779-4472-a24d-59522ebc5a0e-tls-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:29:02.584111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:02.584077    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-88fb69696-qptsg" event={"ID":"42028f3d-7343-45af-a0c7-15c59bf4a40c","Type":"ContainerStarted","Data":"11ed54d41ee11424850b280142088d86afda4db0ade76060096fb861acc22e18"}
Sep 11 17:29:02.584111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:02.584094    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-n2x27"
Sep 11 17:29:02.600666 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:02.600622    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-88fb69696-qptsg" podStartSLOduration=2.018685051 podStartE2EDuration="2.600609104s" totalImagesPullingTime="581.924053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:29:02.599408205 +0000 UTC m=+513.031860980" watchObservedRunningTime="2026-09-11 17:29:02.600609104 +0000 UTC m=+513.033061879"
Sep 11 17:29:02.625827 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:02.625796    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-n2x27"]
Sep 11 17:29:02.633561 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:02.633534    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-22nxx"]
Sep 11 17:29:02.633742 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:02.633720    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-22nxx" podUID="b8215939-16f4-42f1-ad5f-c9adbfb3d532" containerName="authorino" containerID="cri-o://a9f0f8abec37c399c96ddeb86bca3a87272f4e5dc2606500a58b627e7659d255" gracePeriod=30
Sep 11 17:29:02.637869 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:02.637840    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-n2x27"]
Sep 11 17:29:02.858697 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:02.858675    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-22nxx"
Sep 11 17:29:02.881250 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:02.881229    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sp5z\" (UniqueName: \"kubernetes.io/projected/b8215939-16f4-42f1-ad5f-c9adbfb3d532-kube-api-access-4sp5z\") pod \"b8215939-16f4-42f1-ad5f-c9adbfb3d532\" (UID: \"b8215939-16f4-42f1-ad5f-c9adbfb3d532\") "
Sep 11 17:29:02.883524 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:02.883467    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8215939-16f4-42f1-ad5f-c9adbfb3d532-kube-api-access-4sp5z" (OuterVolumeSpecName: "kube-api-access-4sp5z") pod "b8215939-16f4-42f1-ad5f-c9adbfb3d532" (UID: "b8215939-16f4-42f1-ad5f-c9adbfb3d532"). InnerVolumeSpecName "kube-api-access-4sp5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:29:02.982219 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:02.982183    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sp5z\" (UniqueName: \"kubernetes.io/projected/b8215939-16f4-42f1-ad5f-c9adbfb3d532-kube-api-access-4sp5z\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:29:03.589948 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:03.589911    2560 generic.go:356] "Generic (PLEG): container finished" podID="b8215939-16f4-42f1-ad5f-c9adbfb3d532" containerID="a9f0f8abec37c399c96ddeb86bca3a87272f4e5dc2606500a58b627e7659d255" exitCode=0
Sep 11 17:29:03.590389 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:03.589954    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-22nxx"
Sep 11 17:29:03.590389 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:03.589966    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-22nxx" event={"ID":"b8215939-16f4-42f1-ad5f-c9adbfb3d532","Type":"ContainerDied","Data":"a9f0f8abec37c399c96ddeb86bca3a87272f4e5dc2606500a58b627e7659d255"}
Sep 11 17:29:03.590389 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:03.590009    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-22nxx" event={"ID":"b8215939-16f4-42f1-ad5f-c9adbfb3d532","Type":"ContainerDied","Data":"1d2a68d572bace5af09bb80af92019f7d6bab3726bd4ccd48a687ee15d536408"}
Sep 11 17:29:03.590389 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:03.590028    2560 scope.go:117] "RemoveContainer" containerID="a9f0f8abec37c399c96ddeb86bca3a87272f4e5dc2606500a58b627e7659d255"
Sep 11 17:29:03.598396 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:03.598379    2560 scope.go:117] "RemoveContainer" containerID="a9f0f8abec37c399c96ddeb86bca3a87272f4e5dc2606500a58b627e7659d255"
Sep 11 17:29:03.598688 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:29:03.598669    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f0f8abec37c399c96ddeb86bca3a87272f4e5dc2606500a58b627e7659d255\": container with ID starting with a9f0f8abec37c399c96ddeb86bca3a87272f4e5dc2606500a58b627e7659d255 not found: ID does not exist" containerID="a9f0f8abec37c399c96ddeb86bca3a87272f4e5dc2606500a58b627e7659d255"
Sep 11 17:29:03.598765 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:03.598701    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f0f8abec37c399c96ddeb86bca3a87272f4e5dc2606500a58b627e7659d255"} err="failed to get container status \"a9f0f8abec37c399c96ddeb86bca3a87272f4e5dc2606500a58b627e7659d255\": rpc error: code = NotFound desc = could not find container \"a9f0f8abec37c399c96ddeb86bca3a87272f4e5dc2606500a58b627e7659d255\": container with ID starting with a9f0f8abec37c399c96ddeb86bca3a87272f4e5dc2606500a58b627e7659d255 not found: ID does not exist"
Sep 11 17:29:03.609829 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:03.609800    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-22nxx"]
Sep 11 17:29:03.613590 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:03.613571    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-22nxx"]
Sep 11 17:29:04.050303 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:04.050274    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11e1b64a-9779-4472-a24d-59522ebc5a0e" path="/var/lib/kubelet/pods/11e1b64a-9779-4472-a24d-59522ebc5a0e/volumes"
Sep 11 17:29:04.050543 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:04.050529    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8215939-16f4-42f1-ad5f-c9adbfb3d532" path="/var/lib/kubelet/pods/b8215939-16f4-42f1-ad5f-c9adbfb3d532/volumes"
Sep 11 17:29:07.227356 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:07.227324    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:07.289186 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:07.289153    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:07.344761 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:07.344731    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:07.734562 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:07.734531    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:07.832328 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:07.832297    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:11.617961 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:11.617923    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:11.714010 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:11.713975    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:40.950708 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:40.950676    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:41.047410 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:41.047378    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:42.343634 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:42.343601    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:42.444533 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:42.444494    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:42.734897 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:42.734864    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:42.833985 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:42.833950    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:42.926020 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:42.925979    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:52.823224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:52.823189    2560 generic.go:356] "Generic (PLEG): container finished" podID="24d8555f-802e-4533-8e21-d1eb611ba4a6" containerID="6d773a838218ec7dc6f9b15b3fe26547b3808e4a0e0c6a3803d31338b25a74ab" exitCode=0
Sep 11 17:29:52.823640 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:52.823265    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc" event={"ID":"24d8555f-802e-4533-8e21-d1eb611ba4a6","Type":"ContainerDied","Data":"6d773a838218ec7dc6f9b15b3fe26547b3808e4a0e0c6a3803d31338b25a74ab"}
Sep 11 17:29:52.823682 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:52.823643    2560 scope.go:117] "RemoveContainer" containerID="6d773a838218ec7dc6f9b15b3fe26547b3808e4a0e0c6a3803d31338b25a74ab"
Sep 11 17:29:52.824580 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:52.824548    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:29:52.824647 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:52.824635    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:29:52.824682 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:52.824668    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:29:53.833775 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:53.833742    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc" event={"ID":"24d8555f-802e-4533-8e21-d1eb611ba4a6","Type":"ContainerStarted","Data":"7208fb86a4b109a849f1a75984f807db7c38f57b6ed2decebb6548a3ed765c4c"}
Sep 11 17:29:54.309770 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:54.309688    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:54.411607 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:54.411574    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:29:54.411988 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:54.411898    2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wmvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 11 17:29:54.412142 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:54.412119    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc" podUID="24d8555f-802e-4533-8e21-d1eb611ba4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 11 17:29:54.456603 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:54.456573    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:54.650811 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:54.650773    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:29:55.410814 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:55.410776    2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wmvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 11 17:29:55.411501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:55.411451    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc" podUID="24d8555f-802e-4533-8e21-d1eb611ba4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 11 17:29:56.411078 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:56.411038    2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wmvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 11 17:29:56.411555 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:56.411106    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc" podUID="24d8555f-802e-4533-8e21-d1eb611ba4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 11 17:29:57.411636 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:57.411596    2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wmvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 11 17:29:57.412134 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:57.411684    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc" podUID="24d8555f-802e-4533-8e21-d1eb611ba4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 11 17:29:58.411528 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:58.411490    2560 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wmvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 11 17:29:58.411828 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:58.411798    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc" podUID="24d8555f-802e-4533-8e21-d1eb611ba4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 11 17:29:59.414854 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:59.414828    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:29:59.415224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:59.415175    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:29:59.415959 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:29:59.415937    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wmvtc"
Sep 11 17:30:05.064970 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:05.064935    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:30:16.211113 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:16.211072    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:30:28.227987 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.227948    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"]
Sep 11 17:30:28.228436 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.228298    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8215939-16f4-42f1-ad5f-c9adbfb3d532" containerName="authorino"
Sep 11 17:30:28.228436 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.228308    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8215939-16f4-42f1-ad5f-c9adbfb3d532" containerName="authorino"
Sep 11 17:30:28.228436 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.228363    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8215939-16f4-42f1-ad5f-c9adbfb3d532" containerName="authorino"
Sep 11 17:30:28.231357 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.231335    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.234922 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.234900    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 17:30:28.235037 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.234908    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 17:30:28.235037 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.234907    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 17:30:28.236296 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.236274    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-b865k\""
Sep 11 17:30:28.242001 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.241981    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"]
Sep 11 17:30:28.259028 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.259002    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.259147 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.259113    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bxxj\" (UniqueName: \"kubernetes.io/projected/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-kube-api-access-4bxxj\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.259214 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.259151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.259214 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.259181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.259325 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.259215    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.259325 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.259238    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.267395 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.267374    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:30:28.359759 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.359732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bxxj\" (UniqueName: \"kubernetes.io/projected/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-kube-api-access-4bxxj\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.359934 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.359766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.359934 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.359795    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.359934 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.359835    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.359934 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.359857    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.359934 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.359923    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.360247 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.360113    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.360247 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.360229    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.360352 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.360291    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.362076 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.362054    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.362294 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.362275    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.368017 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.367994    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bxxj\" (UniqueName: \"kubernetes.io/projected/100017ae-b96c-4b9b-9e7c-b66f3e2ef228-kube-api-access-4bxxj\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q\" (UID: \"100017ae-b96c-4b9b-9e7c-b66f3e2ef228\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.541989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.541908    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:28.809592 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.809570    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"]
Sep 11 17:30:28.811724 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:30:28.811699    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100017ae_b96c_4b9b_9e7c_b66f3e2ef228.slice/crio-c62fd161973d315a23a58f62b9c43304aaa3a9ee7df54f9c294123d3a2ad7579 WatchSource:0}: Error finding container c62fd161973d315a23a58f62b9c43304aaa3a9ee7df54f9c294123d3a2ad7579: Status 404 returned error can't find the container with id c62fd161973d315a23a58f62b9c43304aaa3a9ee7df54f9c294123d3a2ad7579
Sep 11 17:30:28.991431 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:28.991398    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q" event={"ID":"100017ae-b96c-4b9b-9e7c-b66f3e2ef228","Type":"ContainerStarted","Data":"c62fd161973d315a23a58f62b9c43304aaa3a9ee7df54f9c294123d3a2ad7579"}
Sep 11 17:30:29.993735 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:29.993705    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:30:29.994224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:29.993931    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:30:29.997431 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:29.997404    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q" event={"ID":"100017ae-b96c-4b9b-9e7c-b66f3e2ef228","Type":"ContainerStarted","Data":"1fba39aff9991f67e16b432967313bc549cb0597bfabf9f2ca3a9380c476043c"}
Sep 11 17:30:30.015900 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:30.015844    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q" podStartSLOduration=0.957198551 podStartE2EDuration="2.015810427s" totalImagesPullingTime="1.058611876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:30:30.014672117 +0000 UTC m=+600.447124894" watchObservedRunningTime="2026-09-11 17:30:30.015810427 +0000 UTC m=+600.448263203"
Sep 11 17:30:38.166367 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:38.166330    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:30:38.542358 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:38.542269    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:38.542525 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:38.542361    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:38.554303 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:38.554279    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:39.047409 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:39.047378    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q"
Sep 11 17:30:47.529147 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:47.529117    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:30:57.942749 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:30:57.942663    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:31:07.143015 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.142981    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"]
Sep 11 17:31:07.147198 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.147174    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.150087 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.150064    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 17:31:07.158513 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.158468    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"]
Sep 11 17:31:07.228100 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.228074    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:31:07.308015 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.307986    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.308155 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.308070    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnkjd\" (UniqueName: \"kubernetes.io/projected/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-kube-api-access-tnkjd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.308155 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.308095    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.308155 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.308113    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.308155 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.308129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.308309 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.308193    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.408925 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.408853    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.409066 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.408948    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnkjd\" (UniqueName: \"kubernetes.io/projected/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-kube-api-access-tnkjd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.409066 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.408974    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.409066 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.409001    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.409066 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.409021    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.409066 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.409051    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.409334 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.409311    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.409391 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.409342    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.409456 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.409439    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.410968 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.410943    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.411282 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.411263    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.418353 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.418322    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnkjd\" (UniqueName: \"kubernetes.io/projected/92d4801d-b38f-43e2-b2f7-d996ebbbf7fe-kube-api-access-tnkjd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rmfvj\" (UID: \"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.459170 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.459151    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:07.733336 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.733237    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"]
Sep 11 17:31:07.736111 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:31:07.736086    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d4801d_b38f_43e2_b2f7_d996ebbbf7fe.slice/crio-16bfd3db2e80cdf833a517fb1713035c1e56350e06fa417a210b2312b694fc61 WatchSource:0}: Error finding container 16bfd3db2e80cdf833a517fb1713035c1e56350e06fa417a210b2312b694fc61: Status 404 returned error can't find the container with id 16bfd3db2e80cdf833a517fb1713035c1e56350e06fa417a210b2312b694fc61
Sep 11 17:31:07.737578 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:07.737562    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:31:08.164532 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:08.164497    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj" event={"ID":"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe","Type":"ContainerStarted","Data":"72d70d271bb8c0c9cef3d1d3969eb06b1dd7f71659d492682e0d4e4ead86881f"}
Sep 11 17:31:08.164966 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:08.164548    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj" event={"ID":"92d4801d-b38f-43e2-b2f7-d996ebbbf7fe","Type":"ContainerStarted","Data":"16bfd3db2e80cdf833a517fb1713035c1e56350e06fa417a210b2312b694fc61"}
Sep 11 17:31:08.184589 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:08.184542    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj" podStartSLOduration=0.969346493 podStartE2EDuration="1.184526396s" totalImagesPullingTime="215.179903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:08.182301028 +0000 UTC m=+638.614753809" watchObservedRunningTime="2026-09-11 17:31:08.184526396 +0000 UTC m=+638.616979171"
Sep 11 17:31:14.518755 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.518720    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"]
Sep 11 17:31:14.522312 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.522292    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.525149 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.525121    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 17:31:14.533950 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.533929    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"]
Sep 11 17:31:14.548948 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.548927    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:31:14.578141 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.578114    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b594845d-6db1-42db-a77c-dfbcd6b0db0f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.578263 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.578148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b594845d-6db1-42db-a77c-dfbcd6b0db0f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.578263 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.578170    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b594845d-6db1-42db-a77c-dfbcd6b0db0f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.578263 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.578217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b594845d-6db1-42db-a77c-dfbcd6b0db0f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.578371 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.578263    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmzmj\" (UniqueName: \"kubernetes.io/projected/b594845d-6db1-42db-a77c-dfbcd6b0db0f-kube-api-access-kmzmj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.578371 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.578364    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b594845d-6db1-42db-a77c-dfbcd6b0db0f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.679607 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.679566    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmzmj\" (UniqueName: \"kubernetes.io/projected/b594845d-6db1-42db-a77c-dfbcd6b0db0f-kube-api-access-kmzmj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.679795 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.679691    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b594845d-6db1-42db-a77c-dfbcd6b0db0f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.679795 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.679736    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b594845d-6db1-42db-a77c-dfbcd6b0db0f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.679795 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.679769    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b594845d-6db1-42db-a77c-dfbcd6b0db0f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.679969 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.679796    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b594845d-6db1-42db-a77c-dfbcd6b0db0f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.679969 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.679822    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b594845d-6db1-42db-a77c-dfbcd6b0db0f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.680187 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.680162    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b594845d-6db1-42db-a77c-dfbcd6b0db0f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.680256 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.680173    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b594845d-6db1-42db-a77c-dfbcd6b0db0f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.680297 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.680284    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b594845d-6db1-42db-a77c-dfbcd6b0db0f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.682054 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.682030    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b594845d-6db1-42db-a77c-dfbcd6b0db0f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.682200 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.682181    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b594845d-6db1-42db-a77c-dfbcd6b0db0f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.687032 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.687010    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmzmj\" (UniqueName: \"kubernetes.io/projected/b594845d-6db1-42db-a77c-dfbcd6b0db0f-kube-api-access-kmzmj\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6\" (UID: \"b594845d-6db1-42db-a77c-dfbcd6b0db0f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:14.831595 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:14.831504    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:15.111339 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:15.111316    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"]
Sep 11 17:31:15.113095 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:31:15.113068    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb594845d_6db1_42db_a77c_dfbcd6b0db0f.slice/crio-c7176315df177c564c85dd740bee142982e806a317d3e6d7f4f0938e54cbff25 WatchSource:0}: Error finding container c7176315df177c564c85dd740bee142982e806a317d3e6d7f4f0938e54cbff25: Status 404 returned error can't find the container with id c7176315df177c564c85dd740bee142982e806a317d3e6d7f4f0938e54cbff25
Sep 11 17:31:15.197706 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:15.197672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6" event={"ID":"b594845d-6db1-42db-a77c-dfbcd6b0db0f","Type":"ContainerStarted","Data":"c7176315df177c564c85dd740bee142982e806a317d3e6d7f4f0938e54cbff25"}
Sep 11 17:31:16.203995 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:16.203959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6" event={"ID":"b594845d-6db1-42db-a77c-dfbcd6b0db0f","Type":"ContainerStarted","Data":"7443ddc0c81d01066b95c05e06f86ac14e73a5ff82d1e9f78989741f31dda3eb"}
Sep 11 17:31:16.223114 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:16.223059    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6" podStartSLOduration=1.9975246549999999 podStartE2EDuration="2.223041467s" totalImagesPullingTime="225.516812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:16.222125997 +0000 UTC m=+646.654578775" watchObservedRunningTime="2026-09-11 17:31:16.223041467 +0000 UTC m=+646.655494242"
Sep 11 17:31:17.460032 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:17.459998    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:17.460414 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:17.460072    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:17.472456 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:17.472429    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:18.224527 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:18.224498    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rmfvj"
Sep 11 17:31:24.832109 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:24.832067    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:24.832724 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:24.832132    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:24.844523 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:24.844495    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:25.255230 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:25.255200    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6"
Sep 11 17:31:48.597933 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:48.597896    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:31:52.818505 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:52.818445    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:31:56.326318 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:31:56.326289    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:03.350964 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:03.350926    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64d78656b9-72zss"]
Sep 11 17:32:03.354321 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:03.354302    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64d78656b9-72zss"
Sep 11 17:32:03.365454 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:03.365430    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64d78656b9-72zss"]
Sep 11 17:32:03.433927 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:03.433897    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0975010-1a10-4076-814e-64ce01d14694-tls-cert\") pod \"authorino-64d78656b9-72zss\" (UID: \"b0975010-1a10-4076-814e-64ce01d14694\") " pod="kuadrant-system/authorino-64d78656b9-72zss"
Sep 11 17:32:03.434112 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:03.434034    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwsgf\" (UniqueName: \"kubernetes.io/projected/b0975010-1a10-4076-814e-64ce01d14694-kube-api-access-fwsgf\") pod \"authorino-64d78656b9-72zss\" (UID: \"b0975010-1a10-4076-814e-64ce01d14694\") " pod="kuadrant-system/authorino-64d78656b9-72zss"
Sep 11 17:32:03.535273 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:03.535235    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwsgf\" (UniqueName: \"kubernetes.io/projected/b0975010-1a10-4076-814e-64ce01d14694-kube-api-access-fwsgf\") pod \"authorino-64d78656b9-72zss\" (UID: \"b0975010-1a10-4076-814e-64ce01d14694\") " pod="kuadrant-system/authorino-64d78656b9-72zss"
Sep 11 17:32:03.535513 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:03.535286    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0975010-1a10-4076-814e-64ce01d14694-tls-cert\") pod \"authorino-64d78656b9-72zss\" (UID: \"b0975010-1a10-4076-814e-64ce01d14694\") " pod="kuadrant-system/authorino-64d78656b9-72zss"
Sep 11 17:32:03.537678 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:03.537655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0975010-1a10-4076-814e-64ce01d14694-tls-cert\") pod \"authorino-64d78656b9-72zss\" (UID: \"b0975010-1a10-4076-814e-64ce01d14694\") " pod="kuadrant-system/authorino-64d78656b9-72zss"
Sep 11 17:32:03.545622 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:03.545599    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwsgf\" (UniqueName: \"kubernetes.io/projected/b0975010-1a10-4076-814e-64ce01d14694-kube-api-access-fwsgf\") pod \"authorino-64d78656b9-72zss\" (UID: \"b0975010-1a10-4076-814e-64ce01d14694\") " pod="kuadrant-system/authorino-64d78656b9-72zss"
Sep 11 17:32:03.663989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:03.663950    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64d78656b9-72zss"
Sep 11 17:32:03.944280 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:03.944193    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64d78656b9-72zss"]
Sep 11 17:32:03.948800 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:32:03.948769    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0975010_1a10_4076_814e_64ce01d14694.slice/crio-3bfb09d4ca3c08efdc7655f576327252b372582c3c18ddf33b9348ca41ac601a WatchSource:0}: Error finding container 3bfb09d4ca3c08efdc7655f576327252b372582c3c18ddf33b9348ca41ac601a: Status 404 returned error can't find the container with id 3bfb09d4ca3c08efdc7655f576327252b372582c3c18ddf33b9348ca41ac601a
Sep 11 17:32:04.161312 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:04.161277    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:04.437815 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:04.437780    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64d78656b9-72zss" event={"ID":"b0975010-1a10-4076-814e-64ce01d14694","Type":"ContainerStarted","Data":"3bfb09d4ca3c08efdc7655f576327252b372582c3c18ddf33b9348ca41ac601a"}
Sep 11 17:32:05.444608 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:05.444566    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64d78656b9-72zss" event={"ID":"b0975010-1a10-4076-814e-64ce01d14694","Type":"ContainerStarted","Data":"4a5641b754cbad1095edd16f7922a88570209f8a2dc539cc33516af3f0dc53f2"}
Sep 11 17:32:05.476710 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:05.476647    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-64d78656b9-72zss" podStartSLOduration=1.519798913 podStartE2EDuration="2.476629006s" totalImagesPullingTime="956.830093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:05.474618669 +0000 UTC m=+695.907071451" watchObservedRunningTime="2026-09-11 17:32:05.476629006 +0000 UTC m=+695.909081783"
Sep 11 17:32:05.521767 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:05.521733    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-88fb69696-qptsg"]
Sep 11 17:32:05.522198 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:05.522144    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-88fb69696-qptsg" podUID="42028f3d-7343-45af-a0c7-15c59bf4a40c" containerName="authorino" containerID="cri-o://11ed54d41ee11424850b280142088d86afda4db0ade76060096fb861acc22e18" gracePeriod=30
Sep 11 17:32:05.782346 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:05.782313    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-88fb69696-qptsg"
Sep 11 17:32:05.864203 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:05.864170    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvqhn\" (UniqueName: \"kubernetes.io/projected/42028f3d-7343-45af-a0c7-15c59bf4a40c-kube-api-access-xvqhn\") pod \"42028f3d-7343-45af-a0c7-15c59bf4a40c\" (UID: \"42028f3d-7343-45af-a0c7-15c59bf4a40c\") "
Sep 11 17:32:05.864383 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:05.864281    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/42028f3d-7343-45af-a0c7-15c59bf4a40c-tls-cert\") pod \"42028f3d-7343-45af-a0c7-15c59bf4a40c\" (UID: \"42028f3d-7343-45af-a0c7-15c59bf4a40c\") "
Sep 11 17:32:05.866369 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:05.866339    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42028f3d-7343-45af-a0c7-15c59bf4a40c-kube-api-access-xvqhn" (OuterVolumeSpecName: "kube-api-access-xvqhn") pod "42028f3d-7343-45af-a0c7-15c59bf4a40c" (UID: "42028f3d-7343-45af-a0c7-15c59bf4a40c"). InnerVolumeSpecName "kube-api-access-xvqhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:32:05.875946 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:05.875913    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42028f3d-7343-45af-a0c7-15c59bf4a40c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "42028f3d-7343-45af-a0c7-15c59bf4a40c" (UID: "42028f3d-7343-45af-a0c7-15c59bf4a40c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:32:05.965273 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:05.965227    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvqhn\" (UniqueName: \"kubernetes.io/projected/42028f3d-7343-45af-a0c7-15c59bf4a40c-kube-api-access-xvqhn\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:32:05.965273 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:05.965256    2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/42028f3d-7343-45af-a0c7-15c59bf4a40c-tls-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:32:06.179264 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:06.179231    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:06.361464 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:06.361387    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:06.450988 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:06.450943    2560 generic.go:356] "Generic (PLEG): container finished" podID="42028f3d-7343-45af-a0c7-15c59bf4a40c" containerID="11ed54d41ee11424850b280142088d86afda4db0ade76060096fb861acc22e18" exitCode=0
Sep 11 17:32:06.450988 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:06.450985    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-88fb69696-qptsg"
Sep 11 17:32:06.451556 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:06.450994    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-88fb69696-qptsg" event={"ID":"42028f3d-7343-45af-a0c7-15c59bf4a40c","Type":"ContainerDied","Data":"11ed54d41ee11424850b280142088d86afda4db0ade76060096fb861acc22e18"}
Sep 11 17:32:06.451556 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:06.451034    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-88fb69696-qptsg" event={"ID":"42028f3d-7343-45af-a0c7-15c59bf4a40c","Type":"ContainerDied","Data":"5f5165c50ef02e799af6030db2f832c219c95c4b6769a3b586862390a6ee6d5c"}
Sep 11 17:32:06.451556 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:06.451050    2560 scope.go:117] "RemoveContainer" containerID="11ed54d41ee11424850b280142088d86afda4db0ade76060096fb861acc22e18"
Sep 11 17:32:06.459990 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:06.459967    2560 scope.go:117] "RemoveContainer" containerID="11ed54d41ee11424850b280142088d86afda4db0ade76060096fb861acc22e18"
Sep 11 17:32:06.460302 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:32:06.460276    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ed54d41ee11424850b280142088d86afda4db0ade76060096fb861acc22e18\": container with ID starting with 11ed54d41ee11424850b280142088d86afda4db0ade76060096fb861acc22e18 not found: ID does not exist" containerID="11ed54d41ee11424850b280142088d86afda4db0ade76060096fb861acc22e18"
Sep 11 17:32:06.460397 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:06.460329    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ed54d41ee11424850b280142088d86afda4db0ade76060096fb861acc22e18"} err="failed to get container status \"11ed54d41ee11424850b280142088d86afda4db0ade76060096fb861acc22e18\": rpc error: code = NotFound desc = could not find container \"11ed54d41ee11424850b280142088d86afda4db0ade76060096fb861acc22e18\": container with ID starting with 11ed54d41ee11424850b280142088d86afda4db0ade76060096fb861acc22e18 not found: ID does not exist"
Sep 11 17:32:06.489511 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:06.489463    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-88fb69696-qptsg"]
Sep 11 17:32:06.505265 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:06.505234    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-88fb69696-qptsg"]
Sep 11 17:32:07.477535 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:07.477497    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:07.627166 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:07.627080    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:08.051498 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:08.051441    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42028f3d-7343-45af-a0c7-15c59bf4a40c" path="/var/lib/kubelet/pods/42028f3d-7343-45af-a0c7-15c59bf4a40c/volumes"
Sep 11 17:32:12.923335 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:12.923299    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:13.033938 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:13.033907    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:23.393157 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:23.393081    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:23.528850 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:23.528792    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:23.927764 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:23.927728    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:24.203402 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:24.203317    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:24.813491 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:24.813440    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:24.907559 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:24.907523    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:25.317978 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:25.317937    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:29.978457 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:29.978418    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7738-5be345"]
Sep 11 17:32:29.978844 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:29.978809    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42028f3d-7343-45af-a0c7-15c59bf4a40c" containerName="authorino"
Sep 11 17:32:29.978844 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:29.978822    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="42028f3d-7343-45af-a0c7-15c59bf4a40c" containerName="authorino"
Sep 11 17:32:29.978935 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:29.978897    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="42028f3d-7343-45af-a0c7-15c59bf4a40c" containerName="authorino"
Sep 11 17:32:29.981801 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:29.981784    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7738-5be345"
Sep 11 17:32:29.985163 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:29.985127    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jtt66\""
Sep 11 17:32:29.987209 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:29.987182    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7738-5be345"]
Sep 11 17:32:30.093534 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:30.093465    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svf4s\" (UniqueName: \"kubernetes.io/projected/a30708e1-8e2e-41f7-b4d9-cbd917602253-kube-api-access-svf4s\") pod \"test-curl-7738-5be345\" (UID: \"a30708e1-8e2e-41f7-b4d9-cbd917602253\") " pod="opendatahub/test-curl-7738-5be345"
Sep 11 17:32:30.194147 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:30.194103    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svf4s\" (UniqueName: \"kubernetes.io/projected/a30708e1-8e2e-41f7-b4d9-cbd917602253-kube-api-access-svf4s\") pod \"test-curl-7738-5be345\" (UID: \"a30708e1-8e2e-41f7-b4d9-cbd917602253\") " pod="opendatahub/test-curl-7738-5be345"
Sep 11 17:32:30.203332 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:30.203302    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svf4s\" (UniqueName: \"kubernetes.io/projected/a30708e1-8e2e-41f7-b4d9-cbd917602253-kube-api-access-svf4s\") pod \"test-curl-7738-5be345\" (UID: \"a30708e1-8e2e-41f7-b4d9-cbd917602253\") " pod="opendatahub/test-curl-7738-5be345"
Sep 11 17:32:30.292566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:30.292463    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7738-5be345"
Sep 11 17:32:30.575520 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:30.572281    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7738-5be345"]
Sep 11 17:32:31.167883 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:31.167845    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:31.267420 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:31.267391    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:31.571811 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:31.571766    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7738-5be345" event={"ID":"a30708e1-8e2e-41f7-b4d9-cbd917602253","Type":"ContainerStarted","Data":"a4750496341f7994fbdeeb43f763aec86cba2c7f89d35a9e15b5217388a357f6"}
Sep 11 17:32:31.571811 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:31.571816    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7738-5be345" event={"ID":"a30708e1-8e2e-41f7-b4d9-cbd917602253","Type":"ContainerStarted","Data":"93e58fd3e2c09ae8f299fb7cf1c0b049e040351888079122fc93d8a17efa4a68"}
Sep 11 17:32:31.589597 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:31.589548    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7738-5be345" podStartSLOduration=1.7865406830000001 podStartE2EDuration="2.589533217s" totalImagesPullingTime="802.992534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:31.588389145 +0000 UTC m=+722.020841921" watchObservedRunningTime="2026-09-11 17:32:31.589533217 +0000 UTC m=+722.021985993"
Sep 11 17:32:31.740345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:31.740317    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:31.898453 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:31.898422    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:32.109199 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.109161    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:32.349225 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.349132    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"]
Sep 11 17:32:32.353222 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.353199    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.357697 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.357675    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-40972c14-openshift-default-dockercfg-vmh2p\""
Sep 11 17:32:32.379106 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.379081    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"]
Sep 11 17:32:32.388594 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.388568    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7738-5be345"]
Sep 11 17:32:32.392890 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.392857    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7738-5be345"]
Sep 11 17:32:32.405666 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.405595    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:32.502594 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.502565    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:32.517730 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.517695    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-workload-socket\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.517730 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.517730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03518277-b759-461b-8885-191dd031282a-istio-podinfo\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.517941 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.517758    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03518277-b759-461b-8885-191dd031282a-istiod-ca-cert\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.517941 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.517859    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-istio-envoy\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.517941 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.517890    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spggr\" (UniqueName: \"kubernetes.io/projected/03518277-b759-461b-8885-191dd031282a-kube-api-access-spggr\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.517941 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.517922    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-credential-socket\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.518093 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.517986    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-istio-data\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.518093 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.518043    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-workload-certs\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.518093 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.518060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03518277-b759-461b-8885-191dd031282a-istio-token\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.576795 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:32:32.576768    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7738-5be345" container="test-curl-7738-5be345"
Sep 11 17:32:32.618837 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.618780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-workload-socket\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.618837 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.618806    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03518277-b759-461b-8885-191dd031282a-istio-podinfo\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.618837 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.618824    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03518277-b759-461b-8885-191dd031282a-istiod-ca-cert\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.619046 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.618851    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-istio-envoy\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.619046 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.618868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spggr\" (UniqueName: \"kubernetes.io/projected/03518277-b759-461b-8885-191dd031282a-kube-api-access-spggr\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.619046 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.618896    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-credential-socket\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.619046 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.618938    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-istio-data\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.619046 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.619000    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-workload-certs\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.619046 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.619021    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03518277-b759-461b-8885-191dd031282a-istio-token\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.619420 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.619214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-workload-socket\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.619420 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.619334    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-credential-socket\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.619420 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.619402    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-istio-data\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.619614 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.619545    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-workload-certs\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.619735 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.619713    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03518277-b759-461b-8885-191dd031282a-istiod-ca-cert\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.621056 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.621038    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-istio-envoy\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.621354 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.621338    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03518277-b759-461b-8885-191dd031282a-istio-podinfo\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.628933 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.628901    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03518277-b759-461b-8885-191dd031282a-istio-token\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.629205 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.629177    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spggr\" (UniqueName: \"kubernetes.io/projected/03518277-b759-461b-8885-191dd031282a-kube-api-access-spggr\") pod \"e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65\" (UID: \"03518277-b759-461b-8885-191dd031282a\") " pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.664549 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.664513    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:32.728642 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.728614    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:32.929898 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.929864    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:32.947399 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.947370    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"]
Sep 11 17:32:32.948705 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:32:32.948684    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03518277_b759_461b_8885_191dd031282a.slice/crio-088c993bda4b48fe13bb1a266db2fcbfe01f86dc1be8adee9f8ce4bf1c266109 WatchSource:0}: Error finding container 088c993bda4b48fe13bb1a266db2fcbfe01f86dc1be8adee9f8ce4bf1c266109: Status 404 returned error can't find the container with id 088c993bda4b48fe13bb1a266db2fcbfe01f86dc1be8adee9f8ce4bf1c266109
Sep 11 17:32:32.951507 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.951448    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:32:32.951615 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.951551    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:32:32.951615 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:32.951598    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:32:33.025543 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.025514    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:33.141602 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.141567    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:33.257275 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.257189    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"]
Sep 11 17:32:33.257807 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.257789    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a30708e1-8e2e-41f7-b4d9-cbd917602253" containerName="test-curl-7738-5be345"
Sep 11 17:32:33.257862 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.257812    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a30708e1-8e2e-41f7-b4d9-cbd917602253" containerName="test-curl-7738-5be345"
Sep 11 17:32:33.257900 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.257891    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a30708e1-8e2e-41f7-b4d9-cbd917602253" containerName="test-curl-7738-5be345"
Sep 11 17:32:33.274973 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.274941    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.277554 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.277526    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"]
Sep 11 17:32:33.278038 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.277842    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-3005e5-openshift-default-dockercfg-8jb9m\""
Sep 11 17:32:33.281362 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.281329    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:33.379175 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.379136    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:33.427531 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.427494    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-workload-socket\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.427531 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.427532    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-podinfo\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.427766 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.427566    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-data\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.427766 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.427642    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e169b63-7546-48ce-85e3-5b807c0dc987-istiod-ca-cert\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.427766 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.427706    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-workload-certs\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.427766 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.427731    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-897d9\" (UniqueName: \"kubernetes.io/projected/0e169b63-7546-48ce-85e3-5b807c0dc987-kube-api-access-897d9\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.427974 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.427820    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-token\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.427974 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.427880    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-envoy\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.427974 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.427915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-credential-socket\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.478579 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.478550    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:33.529216 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.529127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-token\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.529216 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.529189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-envoy\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.529456 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.529223    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-credential-socket\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.529456 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.529274    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-workload-socket\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.529456 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.529305    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-podinfo\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.529456 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.529345    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-data\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.529456 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.529382    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e169b63-7546-48ce-85e3-5b807c0dc987-istiod-ca-cert\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.529456 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.529454    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-workload-certs\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.529795 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.529509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-897d9\" (UniqueName: \"kubernetes.io/projected/0e169b63-7546-48ce-85e3-5b807c0dc987-kube-api-access-897d9\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.529795 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.529728    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-credential-socket\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.529795 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.529766    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-data\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.530171 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.530151    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-workload-socket\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.530229 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.530187    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e169b63-7546-48ce-85e3-5b807c0dc987-istiod-ca-cert\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.530284 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.530244    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-workload-certs\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.531810 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.531784    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-envoy\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.532571 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.532547    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-podinfo\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.537787 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.537761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-token\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.537887 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.537838    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-897d9\" (UniqueName: \"kubernetes.io/projected/0e169b63-7546-48ce-85e3-5b807c0dc987-kube-api-access-897d9\") pod \"e2e-autores-3005e5-openshift-default-64f9944598-m8m4s\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") " pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.582850 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.582820    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65" event={"ID":"03518277-b759-461b-8885-191dd031282a","Type":"ContainerStarted","Data":"ab0358d324fd9eb8bc85cc335ef78cc1e4ce995f63f018080046bdc016dd84c9"}
Sep 11 17:32:33.583036 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.582882    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65" event={"ID":"03518277-b759-461b-8885-191dd031282a","Type":"ContainerStarted","Data":"088c993bda4b48fe13bb1a266db2fcbfe01f86dc1be8adee9f8ce4bf1c266109"}
Sep 11 17:32:33.583036 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.582894    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7738-5be345" podUID="a30708e1-8e2e-41f7-b4d9-cbd917602253" containerName="test-curl-7738-5be345" containerID="cri-o://a4750496341f7994fbdeeb43f763aec86cba2c7f89d35a9e15b5217388a357f6" gracePeriod=2
Sep 11 17:32:33.587437 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.587414    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:33.612428 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.612372    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65" podStartSLOduration=1.6123514349999999 podStartE2EDuration="1.612351435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:33.611295369 +0000 UTC m=+724.043748147" watchObservedRunningTime="2026-09-11 17:32:33.612351435 +0000 UTC m=+724.044804214"
Sep 11 17:32:33.613805 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:32:33.613772    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7738-5be345\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="a30708e1-8e2e-41f7-b4d9-cbd917602253" pod="opendatahub/test-curl-7738-5be345"
Sep 11 17:32:33.616085 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:32:33.616057    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7738-5be345\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="a30708e1-8e2e-41f7-b4d9-cbd917602253" pod="opendatahub/test-curl-7738-5be345"
Sep 11 17:32:33.665025 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.664744    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:33.874868 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.874839    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"]
Sep 11 17:32:33.877049 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:32:33.877020    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e169b63_7546_48ce_85e3_5b807c0dc987.slice/crio-3dc58c59e275a6b66dad33a51715ab08a5d33620523860cde08e61e28e6c5a1a WatchSource:0}: Error finding container 3dc58c59e275a6b66dad33a51715ab08a5d33620523860cde08e61e28e6c5a1a: Status 404 returned error can't find the container with id 3dc58c59e275a6b66dad33a51715ab08a5d33620523860cde08e61e28e6c5a1a
Sep 11 17:32:33.880112 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.880074    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:32:33.880216 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.880150    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:32:33.880216 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:33.880181    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:32:34.350017 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:34.349973    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:34.589944 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:34.589910    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s" event={"ID":"0e169b63-7546-48ce-85e3-5b807c0dc987","Type":"ContainerStarted","Data":"cf16fec358971e100113acb5fe5d156432e8b122c03d0f191e304cf620dfd89c"}
Sep 11 17:32:34.590407 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:34.589963    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s" event={"ID":"0e169b63-7546-48ce-85e3-5b807c0dc987","Type":"ContainerStarted","Data":"3dc58c59e275a6b66dad33a51715ab08a5d33620523860cde08e61e28e6c5a1a"}
Sep 11 17:32:34.649161 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:34.649095    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s" podStartSLOduration=1.649072559 podStartE2EDuration="1.649072559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:34.645420745 +0000 UTC m=+725.077873525" watchObservedRunningTime="2026-09-11 17:32:34.649072559 +0000 UTC m=+725.081525338"
Sep 11 17:32:34.671622 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:34.671590    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:35.055802 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:35.055701    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:35.146400 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:35.146363    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:35.335762 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:35.335667    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:35.588217 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:35.588185    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:35.594419 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:35.594396    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:35.595045 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:35.595022    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:35.595208 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:35.595189    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:35.595678 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:35.595657    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:35.595888 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:35.595875    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:35.796932 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:35.796900    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7738-5be345"
Sep 11 17:32:35.854789 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:35.854709    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svf4s\" (UniqueName: \"kubernetes.io/projected/a30708e1-8e2e-41f7-b4d9-cbd917602253-kube-api-access-svf4s\") pod \"a30708e1-8e2e-41f7-b4d9-cbd917602253\" (UID: \"a30708e1-8e2e-41f7-b4d9-cbd917602253\") "
Sep 11 17:32:35.856882 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:35.856850    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30708e1-8e2e-41f7-b4d9-cbd917602253-kube-api-access-svf4s" (OuterVolumeSpecName: "kube-api-access-svf4s") pod "a30708e1-8e2e-41f7-b4d9-cbd917602253" (UID: "a30708e1-8e2e-41f7-b4d9-cbd917602253"). InnerVolumeSpecName "kube-api-access-svf4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:32:35.955570 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:35.955535    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svf4s\" (UniqueName: \"kubernetes.io/projected/a30708e1-8e2e-41f7-b4d9-cbd917602253-kube-api-access-svf4s\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:32:36.051431 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:36.051394    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a30708e1-8e2e-41f7-b4d9-cbd917602253" path="/var/lib/kubelet/pods/a30708e1-8e2e-41f7-b4d9-cbd917602253/volumes"
Sep 11 17:32:36.532533 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:36.532498    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:36.601883 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:36.601849    2560 generic.go:356] "Generic (PLEG): container finished" podID="a30708e1-8e2e-41f7-b4d9-cbd917602253" containerID="a4750496341f7994fbdeeb43f763aec86cba2c7f89d35a9e15b5217388a357f6" exitCode=137
Sep 11 17:32:36.602300 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:36.601907    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7738-5be345"
Sep 11 17:32:36.602300 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:36.601949    2560 scope.go:117] "RemoveContainer" containerID="a4750496341f7994fbdeeb43f763aec86cba2c7f89d35a9e15b5217388a357f6"
Sep 11 17:32:36.610316 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:36.610292    2560 scope.go:117] "RemoveContainer" containerID="a4750496341f7994fbdeeb43f763aec86cba2c7f89d35a9e15b5217388a357f6"
Sep 11 17:32:36.610603 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:32:36.610582    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4750496341f7994fbdeeb43f763aec86cba2c7f89d35a9e15b5217388a357f6\": container with ID starting with a4750496341f7994fbdeeb43f763aec86cba2c7f89d35a9e15b5217388a357f6 not found: ID does not exist" containerID="a4750496341f7994fbdeeb43f763aec86cba2c7f89d35a9e15b5217388a357f6"
Sep 11 17:32:36.610677 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:36.610612    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4750496341f7994fbdeeb43f763aec86cba2c7f89d35a9e15b5217388a357f6"} err="failed to get container status \"a4750496341f7994fbdeeb43f763aec86cba2c7f89d35a9e15b5217388a357f6\": rpc error: code = NotFound desc = could not find container \"a4750496341f7994fbdeeb43f763aec86cba2c7f89d35a9e15b5217388a357f6\": container with ID starting with a4750496341f7994fbdeeb43f763aec86cba2c7f89d35a9e15b5217388a357f6 not found: ID does not exist"
Sep 11 17:32:36.747395 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:36.747359    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:36.956815 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:36.956780    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:37.068039 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:37.068009    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:37.420405 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:37.420371    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:37.518424 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:37.518388    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:38.060805 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:38.060776    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:38.143248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:38.143216    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:38.573940 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:38.573896    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:38.666156 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:38.666125    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:38.746951 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:38.746915    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:39.045699 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:39.045643    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:39.881801 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:39.881772    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:40.737460 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:40.737423    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:41.575808 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:41.575772    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:43.026729 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:43.026700    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:43.200791 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:43.200760    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:43.309901 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:43.309817    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:43.405418 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:43.405386    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:46.139536 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:46.139468    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:48.174945 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:48.174908    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:50.192437 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:50.192411    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:53.425661 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:53.425618    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:54.739965 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:54.739930    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:55.522383 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:55.522343    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:56.164278 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.164246    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:56.326660 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.326580    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:56.543284 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.543246    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:56.727416 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.727378    2560 generic.go:356] "Generic (PLEG): container finished" podID="0e169b63-7546-48ce-85e3-5b807c0dc987" containerID="cf16fec358971e100113acb5fe5d156432e8b122c03d0f191e304cf620dfd89c" exitCode=0
Sep 11 17:32:56.727620 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.727463    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s" event={"ID":"0e169b63-7546-48ce-85e3-5b807c0dc987","Type":"ContainerDied","Data":"cf16fec358971e100113acb5fe5d156432e8b122c03d0f191e304cf620dfd89c"}
Sep 11 17:32:56.728061 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.728038    2560 scope.go:117] "RemoveContainer" containerID="cf16fec358971e100113acb5fe5d156432e8b122c03d0f191e304cf620dfd89c"
Sep 11 17:32:56.729179 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.729154    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:32:56.729234 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.729222    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:32:56.729269 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.729252    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:32:56.838324 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.838235    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"]
Sep 11 17:32:56.842378 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.842242    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:56.846372 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.846341    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d148f3-openshift-default-dockercfg-rhwsb\""
Sep 11 17:32:56.853376 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.853307    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"]
Sep 11 17:32:56.870761 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.870732    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:56.972554 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.972518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-envoy\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:56.972728 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.972564    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-token\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:56.972728 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.972598    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2m6d\" (UniqueName: \"kubernetes.io/projected/a7ee7f20-a157-4d8e-99db-0a37d55febe7-kube-api-access-l2m6d\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:56.972728 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.972616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istiod-ca-cert\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:56.972728 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.972694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-podinfo\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:56.972894 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.972733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-data\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:56.972894 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.972761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-workload-socket\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:56.972894 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.972815    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-workload-certs\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:56.972894 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:56.972856    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-credential-socket\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.074489 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.074397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-envoy\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.074489 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.074444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-token\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.075929 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.074502    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2m6d\" (UniqueName: \"kubernetes.io/projected/a7ee7f20-a157-4d8e-99db-0a37d55febe7-kube-api-access-l2m6d\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.075929 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.074526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istiod-ca-cert\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.075929 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.074562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-podinfo\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.075929 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.074587    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-data\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.075929 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.074614    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-workload-socket\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.075929 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.074646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-workload-certs\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.075929 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.074686    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-credential-socket\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.075929 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.075012    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-data\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.075929 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.075064    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-workload-certs\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.075929 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.075253    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-workload-socket\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.075929 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.075311    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istiod-ca-cert\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.075929 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.075334    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-credential-socket\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.076880 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.076855    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-envoy\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.077404 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.077387    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-podinfo\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.091451 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.091411    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-token\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.094132 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.094099    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2m6d\" (UniqueName: \"kubernetes.io/projected/a7ee7f20-a157-4d8e-99db-0a37d55febe7-kube-api-access-l2m6d\") pod \"e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") " pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.158789 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.158759    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:57.329054 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.328779    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:57.430059 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.429842    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:57.454623 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.454592    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"]
Sep 11 17:32:57.455332 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:32:57.455290    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ee7f20_a157_4d8e_99db_0a37d55febe7.slice/crio-3709e11336f6c0b43340fbed96ec6c7457a9f4ee8641b852cd575a6d9e10f87a WatchSource:0}: Error finding container 3709e11336f6c0b43340fbed96ec6c7457a9f4ee8641b852cd575a6d9e10f87a: Status 404 returned error can't find the container with id 3709e11336f6c0b43340fbed96ec6c7457a9f4ee8641b852cd575a6d9e10f87a
Sep 11 17:32:57.457770 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.457736    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:32:57.457851 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.457804    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:32:57.457851 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.457833    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:32:57.738000 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.737948    2560 generic.go:356] "Generic (PLEG): container finished" podID="03518277-b759-461b-8885-191dd031282a" containerID="ab0358d324fd9eb8bc85cc335ef78cc1e4ce995f63f018080046bdc016dd84c9" exitCode=0
Sep 11 17:32:57.738248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.738043    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65" event={"ID":"03518277-b759-461b-8885-191dd031282a","Type":"ContainerDied","Data":"ab0358d324fd9eb8bc85cc335ef78cc1e4ce995f63f018080046bdc016dd84c9"}
Sep 11 17:32:57.738612 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.738579    2560 scope.go:117] "RemoveContainer" containerID="ab0358d324fd9eb8bc85cc335ef78cc1e4ce995f63f018080046bdc016dd84c9"
Sep 11 17:32:57.740065 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.740025    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:32:57.740211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.740119    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:32:57.740211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.740161    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:32:57.740877 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.740789    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q" event={"ID":"a7ee7f20-a157-4d8e-99db-0a37d55febe7","Type":"ContainerStarted","Data":"9e846049f878608ad44cedf07ef2e2d426bfcc221bd11f238721b2d221ce34ce"}
Sep 11 17:32:57.740877 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.740845    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q" event={"ID":"a7ee7f20-a157-4d8e-99db-0a37d55febe7","Type":"ContainerStarted","Data":"3709e11336f6c0b43340fbed96ec6c7457a9f4ee8641b852cd575a6d9e10f87a"}
Sep 11 17:32:57.744555 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.743958    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s" event={"ID":"0e169b63-7546-48ce-85e3-5b807c0dc987","Type":"ContainerStarted","Data":"a10d89d94a8c9ba6c0ea85a7fffc4445f52092a68bf884be72b986c71c9e9d3f"}
Sep 11 17:32:57.786921 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:57.786868    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q" podStartSLOduration=1.78684924 podStartE2EDuration="1.78684924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:57.784111451 +0000 UTC m=+748.216564251" watchObservedRunningTime="2026-09-11 17:32:57.78684924 +0000 UTC m=+748.219302017"
Sep 11 17:32:58.124336 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:58.124110    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:58.159487 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:58.159438    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:58.164634 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:58.164611    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:58.588381 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:58.588346    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:32:58.588900 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:58.588742    2560 patch_prober.go:28] interesting pod/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 11 17:32:58.588900 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:58.588798    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s" podUID="0e169b63-7546-48ce-85e3-5b807c0dc987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 11 17:32:58.754608 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:58.754557    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65" event={"ID":"03518277-b759-461b-8885-191dd031282a","Type":"ContainerStarted","Data":"5a0c58d6ba79d7cb74b0f6fff659f9b344d23f8056c6d366be8f2be2a289e033"}
Sep 11 17:32:58.755097 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:58.755069    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:58.756128 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:58.756086    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:32:59.026837 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:59.026803    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:59.588308 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:59.588275    2560 patch_prober.go:28] interesting pod/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 11 17:32:59.588514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:59.588338    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s" podUID="0e169b63-7546-48ce-85e3-5b807c0dc987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 11 17:32:59.664997 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:59.664944    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:32:59.665228 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:59.665202    2560 patch_prober.go:28] interesting pod/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 11 17:32:59.665319 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:59.665281    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65" podUID="03518277-b759-461b-8885-191dd031282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 11 17:32:59.849960 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:59.849873    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:32:59.960580 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:32:59.960535    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:00.069383 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.069336    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:00.163884 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.163838    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:00.583655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.583560    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv"]
Sep 11 17:33:00.586197 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.586174    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv"
Sep 11 17:33:00.587993 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.587969    2560 patch_prober.go:28] interesting pod/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 11 17:33:00.588087 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.588039    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s" podUID="0e169b63-7546-48ce-85e3-5b807c0dc987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 11 17:33:00.590725 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.590612    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:33:00.590725 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.590626    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:33:00.600238 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.600215    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv"]
Sep 11 17:33:00.665584 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.665536    2560 patch_prober.go:28] interesting pod/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 11 17:33:00.665780 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.665601    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65" podUID="03518277-b759-461b-8885-191dd031282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 11 17:33:00.725218 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.725116    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/449cb264-72f7-4fe1-803f-9d9731481366-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-40972c14-lb8sv\" (UID: \"449cb264-72f7-4fe1-803f-9d9731481366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv"
Sep 11 17:33:00.826340 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.826294    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/449cb264-72f7-4fe1-803f-9d9731481366-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-40972c14-lb8sv\" (UID: \"449cb264-72f7-4fe1-803f-9d9731481366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv"
Sep 11 17:33:00.827195 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.827166    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/449cb264-72f7-4fe1-803f-9d9731481366-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-40972c14-lb8sv\" (UID: \"449cb264-72f7-4fe1-803f-9d9731481366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv"
Sep 11 17:33:00.897326 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.897284    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv"
Sep 11 17:33:00.956499 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:00.954167    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:01.228998 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:01.228172    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv"]
Sep 11 17:33:01.257496 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:33:01.256689    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449cb264_72f7_4fe1_803f_9d9731481366.slice/crio-6eb4c0d2c81b9772568b6f630b1d6c5dc2fbbbdb8b0a4fa9442ced9959788e81 WatchSource:0}: Error finding container 6eb4c0d2c81b9772568b6f630b1d6c5dc2fbbbdb8b0a4fa9442ced9959788e81: Status 404 returned error can't find the container with id 6eb4c0d2c81b9772568b6f630b1d6c5dc2fbbbdb8b0a4fa9442ced9959788e81
Sep 11 17:33:01.588545 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:01.588388    2560 patch_prober.go:28] interesting pod/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 11 17:33:01.588545 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:01.588441    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s" podUID="0e169b63-7546-48ce-85e3-5b807c0dc987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 11 17:33:01.665120 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:01.664753    2560 patch_prober.go:28] interesting pod/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 11 17:33:01.665120 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:01.664796    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65" podUID="03518277-b759-461b-8885-191dd031282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 11 17:33:01.665120 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:01.665073    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:01.770957 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:01.770920    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv" event={"ID":"449cb264-72f7-4fe1-803f-9d9731481366","Type":"ContainerStarted","Data":"6eb4c0d2c81b9772568b6f630b1d6c5dc2fbbbdb8b0a4fa9442ced9959788e81"}
Sep 11 17:33:01.969703 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:01.969671    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:02.054510 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:02.054264    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:02.367932 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:02.367842    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:02.478386 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:02.478356    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:02.584590 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:02.584561    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:02.587713 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:02.587690    2560 patch_prober.go:28] interesting pod/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 11 17:33:02.587874 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:02.587745    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s" podUID="0e169b63-7546-48ce-85e3-5b807c0dc987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 11 17:33:02.665782 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:02.665745    2560 patch_prober.go:28] interesting pod/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 11 17:33:02.666252 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:02.665849    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65" podUID="03518277-b759-461b-8885-191dd031282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 11 17:33:02.938485 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:02.938376    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:03.594164 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:03.594082    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:33:03.594621 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:03.594595    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:33:03.595346 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:03.595327    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:33:03.665859 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:03.665821    2560 patch_prober.go:28] interesting pod/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Sep 11 17:33:03.666283 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:03.665888    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65" podUID="03518277-b759-461b-8885-191dd031282a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Sep 11 17:33:04.020639 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:04.020577    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:04.668735 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:04.668667    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:33:04.669093 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:04.668931    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:33:04.669790 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:04.669773    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:33:04.816585 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:04.816543    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv" event={"ID":"449cb264-72f7-4fe1-803f-9d9731481366","Type":"ContainerStarted","Data":"a96661fc86d12c0eedba843769766bb2f2f16aaa2eee3a0a415b8c5d06cf97a4"}
Sep 11 17:33:04.835765 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:04.835702    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv" podStartSLOduration=1.7100625040000001 podStartE2EDuration="4.835682643s" totalImagesPullingTime="3.125620139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:33:04.83532784 +0000 UTC m=+755.267780620" watchObservedRunningTime="2026-09-11 17:33:04.835682643 +0000 UTC m=+755.268135422"
Sep 11 17:33:04.878498 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:04.878447    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:05.651450 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:05.651412    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:05.824166 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:05.824129    2560 generic.go:356] "Generic (PLEG): container finished" podID="449cb264-72f7-4fe1-803f-9d9731481366" containerID="a96661fc86d12c0eedba843769766bb2f2f16aaa2eee3a0a415b8c5d06cf97a4" exitCode=0
Sep 11 17:33:05.824597 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:05.824193    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv" event={"ID":"449cb264-72f7-4fe1-803f-9d9731481366","Type":"ContainerDied","Data":"a96661fc86d12c0eedba843769766bb2f2f16aaa2eee3a0a415b8c5d06cf97a4"}
Sep 11 17:33:06.330266 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:06.330230    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:06.971928 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:06.971903    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv"
Sep 11 17:33:07.112840 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:07.112734    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/449cb264-72f7-4fe1-803f-9d9731481366-maas-api-service-ca\") pod \"449cb264-72f7-4fe1-803f-9d9731481366\" (UID: \"449cb264-72f7-4fe1-803f-9d9731481366\") "
Sep 11 17:33:07.113118 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:07.113093    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449cb264-72f7-4fe1-803f-9d9731481366-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "449cb264-72f7-4fe1-803f-9d9731481366" (UID: "449cb264-72f7-4fe1-803f-9d9731481366"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:33:07.214422 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:07.214386    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/449cb264-72f7-4fe1-803f-9d9731481366-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:33:07.836869 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:07.836833    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv" event={"ID":"449cb264-72f7-4fe1-803f-9d9731481366","Type":"ContainerDied","Data":"6eb4c0d2c81b9772568b6f630b1d6c5dc2fbbbdb8b0a4fa9442ced9959788e81"}
Sep 11 17:33:07.837053 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:07.836886    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eb4c0d2c81b9772568b6f630b1d6c5dc2fbbbdb8b0a4fa9442ced9959788e81"
Sep 11 17:33:07.837053 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:07.836843    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv"
Sep 11 17:33:16.362562 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:16.362516    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:17.095541 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:17.095509    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:17.276837 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:17.276804    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:17.361000 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:17.360902    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv"]
Sep 11 17:33:17.363774 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:17.363749    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:17.364109 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:17.364045    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40972c14-lb8sv"]
Sep 11 17:33:17.453640 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:17.453592    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:17.486631 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:17.486590    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"]
Sep 11 17:33:17.488443 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:17.487117    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65" podUID="03518277-b759-461b-8885-191dd031282a" containerName="istio-proxy" containerID="cri-o://5a0c58d6ba79d7cb74b0f6fff659f9b344d23f8056c6d366be8f2be2a289e033" gracePeriod=30
Sep 11 17:33:17.525166 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:17.525130    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:17.927228 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:17.927197    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:18.026789 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.026754    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:18.053596 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.053564    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="449cb264-72f7-4fe1-803f-9d9731481366" path="/var/lib/kubelet/pods/449cb264-72f7-4fe1-803f-9d9731481366/volumes"
Sep 11 17:33:18.159082 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.159051    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:18.582920 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.582882    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"]
Sep 11 17:33:18.583342 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.583323    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449cb264-72f7-4fe1-803f-9d9731481366" containerName="revoke-keys"
Sep 11 17:33:18.583390 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.583344    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="449cb264-72f7-4fe1-803f-9d9731481366" containerName="revoke-keys"
Sep 11 17:33:18.583428 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.583421    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="449cb264-72f7-4fe1-803f-9d9731481366" containerName="revoke-keys"
Sep 11 17:33:18.587467 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.587449    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.591859 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.591839    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b76706e3-gw-openshift-default-dockercfg-xlxgb\""
Sep 11 17:33:18.604486 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.604448    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"]
Sep 11 17:33:18.633909 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.633853    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:18.640158 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.640134    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-data\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.640238 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.640165    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-token\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.640238 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.640187    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc212900-6016-4c2e-af48-5cdcad333b3e-istiod-ca-cert\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.640318 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.640268    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-credential-socket\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.640318 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.640298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-envoy\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.640389 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.640318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-podinfo\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.640389 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.640334    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-workload-certs\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.640465 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.640430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-workload-socket\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.640517 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.640468    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6s78\" (UniqueName: \"kubernetes.io/projected/fc212900-6016-4c2e-af48-5cdcad333b3e-kube-api-access-v6s78\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.741739 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.741701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-credential-socket\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.741939 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.741762    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-envoy\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.741939 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.741784    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-podinfo\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.741939 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.741799    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-workload-certs\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.741939 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.741898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-workload-socket\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.741939 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.741934    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6s78\" (UniqueName: \"kubernetes.io/projected/fc212900-6016-4c2e-af48-5cdcad333b3e-kube-api-access-v6s78\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.742228 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.742001    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-data\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.742228 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.742030    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-token\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.742228 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.742060    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc212900-6016-4c2e-af48-5cdcad333b3e-istiod-ca-cert\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.742401 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.742340    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-workload-socket\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.742457 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.742388    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-workload-certs\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.742812 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.742785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-data\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.742961 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.742784    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-credential-socket\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.742961 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.742790    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc212900-6016-4c2e-af48-5cdcad333b3e-istiod-ca-cert\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.744265 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.744247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-envoy\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.744647 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.744626    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-podinfo\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.750328 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.750298    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-token\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.750676 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.750657    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6s78\" (UniqueName: \"kubernetes.io/projected/fc212900-6016-4c2e-af48-5cdcad333b3e-kube-api-access-v6s78\") pod \"e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") " pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.898099 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.898072    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:18.984799 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:18.984765    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:19.090546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.090506    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:19.183077 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.183052    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"]
Sep 11 17:33:19.186307 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:33:19.186279    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc212900_6016_4c2e_af48_5cdcad333b3e.slice/crio-d837eb42f51a5a4d530de0f6521d4af29287fdf56c9a78ffdc893762d1d54490 WatchSource:0}: Error finding container d837eb42f51a5a4d530de0f6521d4af29287fdf56c9a78ffdc893762d1d54490: Status 404 returned error can't find the container with id d837eb42f51a5a4d530de0f6521d4af29287fdf56c9a78ffdc893762d1d54490
Sep 11 17:33:19.188837 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.188797    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:33:19.188922 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.188893    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:33:19.188966 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.188937    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:33:19.671277 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.671245    2560 patch_prober.go:28] interesting pod/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 17:33:19.671679 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.671303    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65" podUID="03518277-b759-461b-8885-191dd031282a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 17:33:19.762637 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.762604    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:19.899414 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.899385    2560 generic.go:356] "Generic (PLEG): container finished" podID="a7ee7f20-a157-4d8e-99db-0a37d55febe7" containerID="9e846049f878608ad44cedf07ef2e2d426bfcc221bd11f238721b2d221ce34ce" exitCode=0
Sep 11 17:33:19.899580 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.899461    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q" event={"ID":"a7ee7f20-a157-4d8e-99db-0a37d55febe7","Type":"ContainerDied","Data":"9e846049f878608ad44cedf07ef2e2d426bfcc221bd11f238721b2d221ce34ce"}
Sep 11 17:33:19.899972 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.899951    2560 scope.go:117] "RemoveContainer" containerID="9e846049f878608ad44cedf07ef2e2d426bfcc221bd11f238721b2d221ce34ce"
Sep 11 17:33:19.901190 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.901153    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:33:19.901305 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.901204    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7" event={"ID":"fc212900-6016-4c2e-af48-5cdcad333b3e","Type":"ContainerStarted","Data":"5bdd21e3dcf8039cdb515cbea50bfee3684cccb4a5eced04a0599b78469fbd42"}
Sep 11 17:33:19.901305 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.901249    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7" event={"ID":"fc212900-6016-4c2e-af48-5cdcad333b3e","Type":"ContainerStarted","Data":"d837eb42f51a5a4d530de0f6521d4af29287fdf56c9a78ffdc893762d1d54490"}
Sep 11 17:33:19.901305 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.901245    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:33:19.901426 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.901336    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:33:19.951335 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:19.951198    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7" podStartSLOduration=1.9511743959999999 podStartE2EDuration="1.951174396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:33:19.947655992 +0000 UTC m=+770.380108772" watchObservedRunningTime="2026-09-11 17:33:19.951174396 +0000 UTC m=+770.383627175"
Sep 11 17:33:20.699548 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:20.699515    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:20.898458 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:20.898418    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:20.903756 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:20.903730    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:20.913005 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:20.912962    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q" event={"ID":"a7ee7f20-a157-4d8e-99db-0a37d55febe7","Type":"ContainerStarted","Data":"27d3da6ad1d165a4f08714e9c5d88b919384603b6eae776cae5d52abe8881484"}
Sep 11 17:33:20.913208 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:20.913037    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:20.914839 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:20.914805    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:21.159745 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:21.159701    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:33:21.159989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:21.159966    2560 patch_prober.go:28] interesting pod/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 17:33:21.160035 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:21.160015    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q" podUID="a7ee7f20-a157-4d8e-99db-0a37d55febe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 17:33:21.617191 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:21.617096    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:21.717802 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:21.717755    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:22.160234 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.160198    2560 patch_prober.go:28] interesting pod/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 17:33:22.160407 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.160270    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q" podUID="a7ee7f20-a157-4d8e-99db-0a37d55febe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 17:33:22.551451 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.551394    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:22.797968 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.797946    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:33:22.887165 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887131    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spggr\" (UniqueName: \"kubernetes.io/projected/03518277-b759-461b-8885-191dd031282a-kube-api-access-spggr\") pod \"03518277-b759-461b-8885-191dd031282a\" (UID: \"03518277-b759-461b-8885-191dd031282a\") "
Sep 11 17:33:22.887345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887183    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03518277-b759-461b-8885-191dd031282a-istio-token\") pod \"03518277-b759-461b-8885-191dd031282a\" (UID: \"03518277-b759-461b-8885-191dd031282a\") "
Sep 11 17:33:22.887345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887247    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-istio-data\") pod \"03518277-b759-461b-8885-191dd031282a\" (UID: \"03518277-b759-461b-8885-191dd031282a\") "
Sep 11 17:33:22.887345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887266    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-workload-certs\") pod \"03518277-b759-461b-8885-191dd031282a\" (UID: \"03518277-b759-461b-8885-191dd031282a\") "
Sep 11 17:33:22.887345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887285    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-istio-envoy\") pod \"03518277-b759-461b-8885-191dd031282a\" (UID: \"03518277-b759-461b-8885-191dd031282a\") "
Sep 11 17:33:22.887345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887314    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-credential-socket\") pod \"03518277-b759-461b-8885-191dd031282a\" (UID: \"03518277-b759-461b-8885-191dd031282a\") "
Sep 11 17:33:22.887649 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887397    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-workload-socket\") pod \"03518277-b759-461b-8885-191dd031282a\" (UID: \"03518277-b759-461b-8885-191dd031282a\") "
Sep 11 17:33:22.887649 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887429    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03518277-b759-461b-8885-191dd031282a-istio-podinfo\") pod \"03518277-b759-461b-8885-191dd031282a\" (UID: \"03518277-b759-461b-8885-191dd031282a\") "
Sep 11 17:33:22.887649 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887454    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03518277-b759-461b-8885-191dd031282a-istiod-ca-cert\") pod \"03518277-b759-461b-8885-191dd031282a\" (UID: \"03518277-b759-461b-8885-191dd031282a\") "
Sep 11 17:33:22.887649 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887559    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "03518277-b759-461b-8885-191dd031282a" (UID: "03518277-b759-461b-8885-191dd031282a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:33:22.887649 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887574    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-istio-data" (OuterVolumeSpecName: "istio-data") pod "03518277-b759-461b-8885-191dd031282a" (UID: "03518277-b759-461b-8885-191dd031282a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:33:22.887918 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887647    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "03518277-b759-461b-8885-191dd031282a" (UID: "03518277-b759-461b-8885-191dd031282a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:33:22.887918 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887716    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "03518277-b759-461b-8885-191dd031282a" (UID: "03518277-b759-461b-8885-191dd031282a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:33:22.887918 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887910    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-istio-data\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:33:22.888056 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887930    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-workload-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:33:22.888056 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887940    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03518277-b759-461b-8885-191dd031282a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "03518277-b759-461b-8885-191dd031282a" (UID: "03518277-b759-461b-8885-191dd031282a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:33:22.888056 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887945    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-credential-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:33:22.888056 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.887960    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-workload-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:33:22.890011 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.889982    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03518277-b759-461b-8885-191dd031282a-kube-api-access-spggr" (OuterVolumeSpecName: "kube-api-access-spggr") pod "03518277-b759-461b-8885-191dd031282a" (UID: "03518277-b759-461b-8885-191dd031282a"). InnerVolumeSpecName "kube-api-access-spggr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:33:22.890011 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.890007    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03518277-b759-461b-8885-191dd031282a-istio-token" (OuterVolumeSpecName: "istio-token") pod "03518277-b759-461b-8885-191dd031282a" (UID: "03518277-b759-461b-8885-191dd031282a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:33:22.890160 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.890006    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "03518277-b759-461b-8885-191dd031282a" (UID: "03518277-b759-461b-8885-191dd031282a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:33:22.890160 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.890006    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03518277-b759-461b-8885-191dd031282a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "03518277-b759-461b-8885-191dd031282a" (UID: "03518277-b759-461b-8885-191dd031282a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:33:22.933223 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.933193    2560 generic.go:356] "Generic (PLEG): container finished" podID="03518277-b759-461b-8885-191dd031282a" containerID="5a0c58d6ba79d7cb74b0f6fff659f9b344d23f8056c6d366be8f2be2a289e033" exitCode=0
Sep 11 17:33:22.933395 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.933237    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"
Sep 11 17:33:22.933395 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.933282    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65" event={"ID":"03518277-b759-461b-8885-191dd031282a","Type":"ContainerDied","Data":"5a0c58d6ba79d7cb74b0f6fff659f9b344d23f8056c6d366be8f2be2a289e033"}
Sep 11 17:33:22.933395 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.933361    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65" event={"ID":"03518277-b759-461b-8885-191dd031282a","Type":"ContainerDied","Data":"088c993bda4b48fe13bb1a266db2fcbfe01f86dc1be8adee9f8ce4bf1c266109"}
Sep 11 17:33:22.933395 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.933388    2560 scope.go:117] "RemoveContainer" containerID="5a0c58d6ba79d7cb74b0f6fff659f9b344d23f8056c6d366be8f2be2a289e033"
Sep 11 17:33:22.944021 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.944002    2560 scope.go:117] "RemoveContainer" containerID="ab0358d324fd9eb8bc85cc335ef78cc1e4ce995f63f018080046bdc016dd84c9"
Sep 11 17:33:22.952086 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.952060    2560 scope.go:117] "RemoveContainer" containerID="5a0c58d6ba79d7cb74b0f6fff659f9b344d23f8056c6d366be8f2be2a289e033"
Sep 11 17:33:22.952396 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:33:22.952372    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0c58d6ba79d7cb74b0f6fff659f9b344d23f8056c6d366be8f2be2a289e033\": container with ID starting with 5a0c58d6ba79d7cb74b0f6fff659f9b344d23f8056c6d366be8f2be2a289e033 not found: ID does not exist" containerID="5a0c58d6ba79d7cb74b0f6fff659f9b344d23f8056c6d366be8f2be2a289e033"
Sep 11 17:33:22.952501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.952409    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0c58d6ba79d7cb74b0f6fff659f9b344d23f8056c6d366be8f2be2a289e033"} err="failed to get container status \"5a0c58d6ba79d7cb74b0f6fff659f9b344d23f8056c6d366be8f2be2a289e033\": rpc error: code = NotFound desc = could not find container \"5a0c58d6ba79d7cb74b0f6fff659f9b344d23f8056c6d366be8f2be2a289e033\": container with ID starting with 5a0c58d6ba79d7cb74b0f6fff659f9b344d23f8056c6d366be8f2be2a289e033 not found: ID does not exist"
Sep 11 17:33:22.952501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.952433    2560 scope.go:117] "RemoveContainer" containerID="ab0358d324fd9eb8bc85cc335ef78cc1e4ce995f63f018080046bdc016dd84c9"
Sep 11 17:33:22.952725 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:33:22.952701    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0358d324fd9eb8bc85cc335ef78cc1e4ce995f63f018080046bdc016dd84c9\": container with ID starting with ab0358d324fd9eb8bc85cc335ef78cc1e4ce995f63f018080046bdc016dd84c9 not found: ID does not exist" containerID="ab0358d324fd9eb8bc85cc335ef78cc1e4ce995f63f018080046bdc016dd84c9"
Sep 11 17:33:22.952771 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.952725    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0358d324fd9eb8bc85cc335ef78cc1e4ce995f63f018080046bdc016dd84c9"} err="failed to get container status \"ab0358d324fd9eb8bc85cc335ef78cc1e4ce995f63f018080046bdc016dd84c9\": rpc error: code = NotFound desc = could not find container \"ab0358d324fd9eb8bc85cc335ef78cc1e4ce995f63f018080046bdc016dd84c9\": container with ID starting with ab0358d324fd9eb8bc85cc335ef78cc1e4ce995f63f018080046bdc016dd84c9 not found: ID does not exist"
Sep 11 17:33:22.959720 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.959687    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"]
Sep 11 17:33:22.964449 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.964426    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-40972c14-openshift-default-6c4d84654d-f2t65"]
Sep 11 17:33:22.989644 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.989625    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03518277-b759-461b-8885-191dd031282a-istio-token\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:33:22.989736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.989647    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03518277-b759-461b-8885-191dd031282a-istio-envoy\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:33:22.989736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.989661    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03518277-b759-461b-8885-191dd031282a-istio-podinfo\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:33:22.989736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.989672    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03518277-b759-461b-8885-191dd031282a-istiod-ca-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:33:22.989736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:22.989687    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spggr\" (UniqueName: \"kubernetes.io/projected/03518277-b759-461b-8885-191dd031282a-kube-api-access-spggr\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:33:23.160183 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:23.160149    2560 patch_prober.go:28] interesting pod/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 17:33:23.160353 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:23.160200    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q" podUID="a7ee7f20-a157-4d8e-99db-0a37d55febe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 17:33:23.657150 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:23.657116    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:24.051969 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:24.051885    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03518277-b759-461b-8885-191dd031282a" path="/var/lib/kubelet/pods/03518277-b759-461b-8885-191dd031282a/volumes"
Sep 11 17:33:24.159749 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:24.159707    2560 patch_prober.go:28] interesting pod/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 17:33:24.159926 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:24.159771    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q" podUID="a7ee7f20-a157-4d8e-99db-0a37d55febe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 17:33:24.436229 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:24.436194    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:24.637884 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:24.637850    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:25.160120 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:25.160084    2560 patch_prober.go:28] interesting pod/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 11 17:33:25.160534 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:25.160142    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q" podUID="a7ee7f20-a157-4d8e-99db-0a37d55febe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 11 17:33:25.337463 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:25.337433    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:25.436705 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:25.436604    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:25.534091 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:25.534058    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:25.850719 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:25.850644    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:25.953021 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:25.952986    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:26.347614 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:26.347569    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:26.377583 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:26.377542    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:33:26.377986 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:26.377956    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:33:26.380000 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:26.379970    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:33:27.138164 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:27.138131    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:34.884072 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:34.884029    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:34.987683 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:34.987646    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:35.095032 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:35.094995    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:40.811107 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:40.811072    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:40.875214 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:40.875182    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:40.961589 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:40.961510    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:41.069052 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:41.068981    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:41.159828 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:41.159785    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:44.059708 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:44.059671    2560 generic.go:356] "Generic (PLEG): container finished" podID="fc212900-6016-4c2e-af48-5cdcad333b3e" containerID="5bdd21e3dcf8039cdb515cbea50bfee3684cccb4a5eced04a0599b78469fbd42" exitCode=0
Sep 11 17:33:44.060279 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:44.059741    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7" event={"ID":"fc212900-6016-4c2e-af48-5cdcad333b3e","Type":"ContainerDied","Data":"5bdd21e3dcf8039cdb515cbea50bfee3684cccb4a5eced04a0599b78469fbd42"}
Sep 11 17:33:44.060279 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:44.060125    2560 scope.go:117] "RemoveContainer" containerID="5bdd21e3dcf8039cdb515cbea50bfee3684cccb4a5eced04a0599b78469fbd42"
Sep 11 17:33:44.061395 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:44.061356    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:33:44.061537 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:44.061456    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:33:44.061616 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:44.061542    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:33:45.066659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:45.066625    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7" event={"ID":"fc212900-6016-4c2e-af48-5cdcad333b3e","Type":"ContainerStarted","Data":"561c0dc70748823e550cfecb72f96a0b7fc2024a0b7965cf25215b826dab55f2"}
Sep 11 17:33:45.898864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:45.898828    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:45.899062 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:45.899038    2560 patch_prober.go:28] interesting pod/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 17:33:45.899113 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:45.899088    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7" podUID="fc212900-6016-4c2e-af48-5cdcad333b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 17:33:46.899180 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:46.899142    2560 patch_prober.go:28] interesting pod/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 17:33:46.899619 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:46.899246    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7" podUID="fc212900-6016-4c2e-af48-5cdcad333b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 17:33:47.570790 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.570734    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:47.642716 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.642682    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:47.745811 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.745776    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:47.898527 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.898457    2560 patch_prober.go:28] interesting pod/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 17:33:47.898718 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.898545    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7" podUID="fc212900-6016-4c2e-af48-5cdcad333b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 17:33:47.941134 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.941096    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"]
Sep 11 17:33:47.941801 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.941779    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03518277-b759-461b-8885-191dd031282a" containerName="istio-proxy"
Sep 11 17:33:47.941864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.941805    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="03518277-b759-461b-8885-191dd031282a" containerName="istio-proxy"
Sep 11 17:33:47.941864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.941822    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03518277-b759-461b-8885-191dd031282a" containerName="istio-proxy"
Sep 11 17:33:47.941864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.941831    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="03518277-b759-461b-8885-191dd031282a" containerName="istio-proxy"
Sep 11 17:33:47.941964 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.941931    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="03518277-b759-461b-8885-191dd031282a" containerName="istio-proxy"
Sep 11 17:33:47.941964 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.941948    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="03518277-b759-461b-8885-191dd031282a" containerName="istio-proxy"
Sep 11 17:33:47.944425 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.944405    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:47.947551 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.947530    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-399fe55a-kserve-self-signed-certs\""
Sep 11 17:33:47.953150 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.953124    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"]
Sep 11 17:33:47.967544 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:47.967511    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:48.068552 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.068511    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-model-cache\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.068735 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.068632    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-dshm\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.068735 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.068680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2896948-942e-40cd-a2ed-a01dbe5944af-tls-certs\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.068810 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.068751    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-home\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.068810 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.068788    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-tmp-dir\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.068881 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.068808    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zr84\" (UniqueName: \"kubernetes.io/projected/b2896948-942e-40cd-a2ed-a01dbe5944af-kube-api-access-4zr84\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.169603 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.169505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2896948-942e-40cd-a2ed-a01dbe5944af-tls-certs\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.169788 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.169610    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-home\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.169788 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.169638    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-tmp-dir\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.169946 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.169920    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zr84\" (UniqueName: \"kubernetes.io/projected/b2896948-942e-40cd-a2ed-a01dbe5944af-kube-api-access-4zr84\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.170081 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.170060    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-home\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.170162 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.170075    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-tmp-dir\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.170162 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.170076    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-model-cache\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.170268 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.170196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-dshm\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.170429 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.170408    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-model-cache\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.172208 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.172185    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2896948-942e-40cd-a2ed-a01dbe5944af-tls-certs\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.172296 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.172189    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-dshm\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.178512 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.178463    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zr84\" (UniqueName: \"kubernetes.io/projected/b2896948-942e-40cd-a2ed-a01dbe5944af-kube-api-access-4zr84\") pod \"e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") " pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.258529 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.258466    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:48.550144 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.550116    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"]
Sep 11 17:33:48.551830 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:33:48.551799    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2896948_942e_40cd_a2ed_a01dbe5944af.slice/crio-96647070e2a45ff5d22a9f94ce3e8cf0f69973b2daed482f4ca8098df410f797 WatchSource:0}: Error finding container 96647070e2a45ff5d22a9f94ce3e8cf0f69973b2daed482f4ca8098df410f797: Status 404 returned error can't find the container with id 96647070e2a45ff5d22a9f94ce3e8cf0f69973b2daed482f4ca8098df410f797
Sep 11 17:33:48.629832 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.629772    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:48.898872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.898836    2560 patch_prober.go:28] interesting pod/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 17:33:48.899041 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:48.898901    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7" podUID="fc212900-6016-4c2e-af48-5cdcad333b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 17:33:49.088070 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:49.088036    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858" event={"ID":"b2896948-942e-40cd-a2ed-a01dbe5944af","Type":"ContainerStarted","Data":"d0050eea5e5e7ecf3923a4d8a82911a7e5e41f01c8d232d245591c616a039484"}
Sep 11 17:33:49.088459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:49.088092    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858" event={"ID":"b2896948-942e-40cd-a2ed-a01dbe5944af","Type":"ContainerStarted","Data":"96647070e2a45ff5d22a9f94ce3e8cf0f69973b2daed482f4ca8098df410f797"}
Sep 11 17:33:49.544807 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:49.544769    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:49.898716 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:49.898679    2560 patch_prober.go:28] interesting pod/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 11 17:33:49.898884 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:49.898739    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7" podUID="fc212900-6016-4c2e-af48-5cdcad333b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 11 17:33:50.278765 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:50.278680    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:50.913691 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:50.913635    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:50.964877 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:50.964836    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:50.986663 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:50.986611    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858" podStartSLOduration=3.986591756 podStartE2EDuration="3.986591756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:33:49.109776873 +0000 UTC m=+799.542229651" watchObservedRunningTime="2026-09-11 17:33:50.986591756 +0000 UTC m=+801.419044572"
Sep 11 17:33:51.099307 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:51.099280    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:33:51.142491 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:51.142424    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:51.423218 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:51.423183    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv"]
Sep 11 17:33:51.425832 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:51.425816    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv"
Sep 11 17:33:51.430097 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:51.430070    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:33:51.430317 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:51.430295    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:33:51.440338 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:51.440311    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv"]
Sep 11 17:33:51.505500 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:51.505316    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51a2af3-27fb-4a41-bc3f-312954b9acf5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv\" (UID: \"f51a2af3-27fb-4a41-bc3f-312954b9acf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv"
Sep 11 17:33:51.606782 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:51.606734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51a2af3-27fb-4a41-bc3f-312954b9acf5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv\" (UID: \"f51a2af3-27fb-4a41-bc3f-312954b9acf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv"
Sep 11 17:33:51.607578 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:51.607556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51a2af3-27fb-4a41-bc3f-312954b9acf5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv\" (UID: \"f51a2af3-27fb-4a41-bc3f-312954b9acf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv"
Sep 11 17:33:51.737801 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:51.737719    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv"
Sep 11 17:33:52.019358 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:52.019333    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv"]
Sep 11 17:33:52.022113 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:33:52.022051    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51a2af3_27fb_4a41_bc3f_312954b9acf5.slice/crio-6c297913e2f4338b6ab23b106dca65cb99d176b850267e667eed3246b7ac16da WatchSource:0}: Error finding container 6c297913e2f4338b6ab23b106dca65cb99d176b850267e667eed3246b7ac16da: Status 404 returned error can't find the container with id 6c297913e2f4338b6ab23b106dca65cb99d176b850267e667eed3246b7ac16da
Sep 11 17:33:52.105311 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:52.105226    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv" event={"ID":"f51a2af3-27fb-4a41-bc3f-312954b9acf5","Type":"ContainerStarted","Data":"6c297913e2f4338b6ab23b106dca65cb99d176b850267e667eed3246b7ac16da"}
Sep 11 17:33:52.148864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:52.148824    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:53.113736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:53.113634    2560 generic.go:356] "Generic (PLEG): container finished" podID="f51a2af3-27fb-4a41-bc3f-312954b9acf5" containerID="c7e3684650a80ff7c9331ec6d7cb9e74df458dac16ff3ad02d25c9ac001c8dd3" exitCode=0
Sep 11 17:33:53.113736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:53.113718    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv" event={"ID":"f51a2af3-27fb-4a41-bc3f-312954b9acf5","Type":"ContainerDied","Data":"c7e3684650a80ff7c9331ec6d7cb9e74df458dac16ff3ad02d25c9ac001c8dd3"}
Sep 11 17:33:53.151423 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:53.151385    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:54.040275 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:54.040246    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:54.262681 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:54.262651    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv"
Sep 11 17:33:54.335739 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:54.335634    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51a2af3-27fb-4a41-bc3f-312954b9acf5-maas-api-service-ca\") pod \"f51a2af3-27fb-4a41-bc3f-312954b9acf5\" (UID: \"f51a2af3-27fb-4a41-bc3f-312954b9acf5\") "
Sep 11 17:33:54.336006 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:54.335984    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51a2af3-27fb-4a41-bc3f-312954b9acf5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f51a2af3-27fb-4a41-bc3f-312954b9acf5" (UID: "f51a2af3-27fb-4a41-bc3f-312954b9acf5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:33:54.437465 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:54.437424    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51a2af3-27fb-4a41-bc3f-312954b9acf5-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:33:54.741869 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:54.741840    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:55.126686 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:55.126599    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv"
Sep 11 17:33:55.126686 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:55.126614    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv" event={"ID":"f51a2af3-27fb-4a41-bc3f-312954b9acf5","Type":"ContainerDied","Data":"6c297913e2f4338b6ab23b106dca65cb99d176b850267e667eed3246b7ac16da"}
Sep 11 17:33:55.126686 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:55.126660    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c297913e2f4338b6ab23b106dca65cb99d176b850267e667eed3246b7ac16da"
Sep 11 17:33:55.534530 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:55.534499    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:33:58.259323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:58.259261    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:58.259323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:58.259329    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:58.272065 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:58.272042    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:33:59.157761 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:33:59.157727    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:34:02.411962 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.411908    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:02.513899 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.513864    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"]
Sep 11 17:34:02.514531 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.514468    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f51a2af3-27fb-4a41-bc3f-312954b9acf5" containerName="revoke-keys"
Sep 11 17:34:02.514659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.514548    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f51a2af3-27fb-4a41-bc3f-312954b9acf5" containerName="revoke-keys"
Sep 11 17:34:02.514783 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.514764    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f51a2af3-27fb-4a41-bc3f-312954b9acf5" containerName="revoke-keys"
Sep 11 17:34:02.515766 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.515745    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:02.516924 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.516906    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.520758 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.520739    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-3005e5\"/\"default-dockercfg-dw76g\""
Sep 11 17:34:02.520859 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.520778    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-3005e5\"/\"kube-root-ca.crt\""
Sep 11 17:34:02.520859 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.520788    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-3005e5\"/\"autores-model-3005e5-kserve-self-signed-certs\""
Sep 11 17:34:02.520859 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.520808    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-3005e5\"/\"openshift-service-ca.crt\""
Sep 11 17:34:02.528238 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.528214    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"]
Sep 11 17:34:02.580821 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.580786    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:02.627675 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.627644    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-tmp-dir\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.627850 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.627706    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-dshm\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.627850 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.627771    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8dc\" (UniqueName: \"kubernetes.io/projected/8fb46243-07f6-406f-ab89-ecb138f0a1e7-kube-api-access-bw8dc\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.627850 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.627803    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb46243-07f6-406f-ab89-ecb138f0a1e7-tls-certs\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.627956 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.627878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-model-cache\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.627956 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.627915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-home\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.707271 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.707181    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:02.728680 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.728645    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-tmp-dir\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.728884 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.728707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-dshm\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.728884 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.728741    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8dc\" (UniqueName: \"kubernetes.io/projected/8fb46243-07f6-406f-ab89-ecb138f0a1e7-kube-api-access-bw8dc\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.728884 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.728768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb46243-07f6-406f-ab89-ecb138f0a1e7-tls-certs\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.728884 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.728837    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-model-cache\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.728884 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.728866    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-home\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.729170 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.729097    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-tmp-dir\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.729226 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.729206    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-home\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.729391 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.729369    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-model-cache\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.730674 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.730649    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv"]
Sep 11 17:34:02.731371 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.731350    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-dshm\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.731824 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.731794    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb46243-07f6-406f-ab89-ecb138f0a1e7-tls-certs\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.734674 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.734634    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b76706e3-nlnjv"]
Sep 11 17:34:02.736947 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.736928    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8dc\" (UniqueName: \"kubernetes.io/projected/8fb46243-07f6-406f-ab89-ecb138f0a1e7-kube-api-access-bw8dc\") pod \"autores-model-3005e5-kserve-578cd9d4c9-pz5gn\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") " pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.800051 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.800018    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:02.827346 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.827314    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:02.876422 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.876389    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"]
Sep 11 17:34:02.876682 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.876654    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7" podUID="fc212900-6016-4c2e-af48-5cdcad333b3e" containerName="istio-proxy" containerID="cri-o://561c0dc70748823e550cfecb72f96a0b7fc2024a0b7965cf25215b826dab55f2" gracePeriod=30
Sep 11 17:34:02.886855 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.886769    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:02.988900 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:02.988801    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:03.110994 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:03.110965    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"]
Sep 11 17:34:03.112839 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:34:03.112814    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fb46243_07f6_406f_ab89_ecb138f0a1e7.slice/crio-fb8308561ab7c34102a28e61ec1373565ab07552bb0f7be4fe27f506af83d354 WatchSource:0}: Error finding container fb8308561ab7c34102a28e61ec1373565ab07552bb0f7be4fe27f506af83d354: Status 404 returned error can't find the container with id fb8308561ab7c34102a28e61ec1373565ab07552bb0f7be4fe27f506af83d354
Sep 11 17:34:03.170440 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:03.170408    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn" event={"ID":"8fb46243-07f6-406f-ab89-ecb138f0a1e7","Type":"ContainerStarted","Data":"fb8308561ab7c34102a28e61ec1373565ab07552bb0f7be4fe27f506af83d354"}
Sep 11 17:34:03.208619 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:03.208582    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:03.318877 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:03.318790    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:03.633425 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:03.633393    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:03.979028 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:03.978944    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:04.054221 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:04.054178    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f51a2af3-27fb-4a41-bc3f-312954b9acf5" path="/var/lib/kubelet/pods/f51a2af3-27fb-4a41-bc3f-312954b9acf5/volumes"
Sep 11 17:34:04.105466 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:04.105430    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:04.176971 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:04.176937    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn" event={"ID":"8fb46243-07f6-406f-ab89-ecb138f0a1e7","Type":"ContainerStarted","Data":"b5db9b7f70bc91635182b2ab4b8e47748b0b57faf32deb3445c77b44f07bf571"}
Sep 11 17:34:04.196448 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:04.196400    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn" podStartSLOduration=2.196383112 podStartE2EDuration="2.196383112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:34:04.195040733 +0000 UTC m=+814.627493513" watchObservedRunningTime="2026-09-11 17:34:04.196383112 +0000 UTC m=+814.628835889"
Sep 11 17:34:04.656901 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:04.656871    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:05.525174 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:05.525135    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:06.100392 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:06.100354    2560 patch_prober.go:28] interesting pod/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 17:34:06.100785 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:06.100423    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7" podUID="fc212900-6016-4c2e-af48-5cdcad333b3e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 17:34:06.255592 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:06.255555    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:06.443907 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:06.443873    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:07.236603 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:07.236569    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:08.149431 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.149401    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:08.162197 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.162173    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:34:08.200322 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.200283    2560 generic.go:356] "Generic (PLEG): container finished" podID="fc212900-6016-4c2e-af48-5cdcad333b3e" containerID="561c0dc70748823e550cfecb72f96a0b7fc2024a0b7965cf25215b826dab55f2" exitCode=0
Sep 11 17:34:08.200638 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.200325    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7" event={"ID":"fc212900-6016-4c2e-af48-5cdcad333b3e","Type":"ContainerDied","Data":"561c0dc70748823e550cfecb72f96a0b7fc2024a0b7965cf25215b826dab55f2"}
Sep 11 17:34:08.200638 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.200632    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7" event={"ID":"fc212900-6016-4c2e-af48-5cdcad333b3e","Type":"ContainerDied","Data":"d837eb42f51a5a4d530de0f6521d4af29287fdf56c9a78ffdc893762d1d54490"}
Sep 11 17:34:08.200803 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.200670    2560 scope.go:117] "RemoveContainer" containerID="561c0dc70748823e550cfecb72f96a0b7fc2024a0b7965cf25215b826dab55f2"
Sep 11 17:34:08.200803 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.200401    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"
Sep 11 17:34:08.214359 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.214330    2560 scope.go:117] "RemoveContainer" containerID="5bdd21e3dcf8039cdb515cbea50bfee3684cccb4a5eced04a0599b78469fbd42"
Sep 11 17:34:08.223402 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.223378    2560 scope.go:117] "RemoveContainer" containerID="561c0dc70748823e550cfecb72f96a0b7fc2024a0b7965cf25215b826dab55f2"
Sep 11 17:34:08.223843 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:34:08.223821    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561c0dc70748823e550cfecb72f96a0b7fc2024a0b7965cf25215b826dab55f2\": container with ID starting with 561c0dc70748823e550cfecb72f96a0b7fc2024a0b7965cf25215b826dab55f2 not found: ID does not exist" containerID="561c0dc70748823e550cfecb72f96a0b7fc2024a0b7965cf25215b826dab55f2"
Sep 11 17:34:08.223907 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.223855    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561c0dc70748823e550cfecb72f96a0b7fc2024a0b7965cf25215b826dab55f2"} err="failed to get container status \"561c0dc70748823e550cfecb72f96a0b7fc2024a0b7965cf25215b826dab55f2\": rpc error: code = NotFound desc = could not find container \"561c0dc70748823e550cfecb72f96a0b7fc2024a0b7965cf25215b826dab55f2\": container with ID starting with 561c0dc70748823e550cfecb72f96a0b7fc2024a0b7965cf25215b826dab55f2 not found: ID does not exist"
Sep 11 17:34:08.223907 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.223874    2560 scope.go:117] "RemoveContainer" containerID="5bdd21e3dcf8039cdb515cbea50bfee3684cccb4a5eced04a0599b78469fbd42"
Sep 11 17:34:08.224189 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:34:08.224161    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bdd21e3dcf8039cdb515cbea50bfee3684cccb4a5eced04a0599b78469fbd42\": container with ID starting with 5bdd21e3dcf8039cdb515cbea50bfee3684cccb4a5eced04a0599b78469fbd42 not found: ID does not exist" containerID="5bdd21e3dcf8039cdb515cbea50bfee3684cccb4a5eced04a0599b78469fbd42"
Sep 11 17:34:08.224246 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.224199    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bdd21e3dcf8039cdb515cbea50bfee3684cccb4a5eced04a0599b78469fbd42"} err="failed to get container status \"5bdd21e3dcf8039cdb515cbea50bfee3684cccb4a5eced04a0599b78469fbd42\": rpc error: code = NotFound desc = could not find container \"5bdd21e3dcf8039cdb515cbea50bfee3684cccb4a5eced04a0599b78469fbd42\": container with ID starting with 5bdd21e3dcf8039cdb515cbea50bfee3684cccb4a5eced04a0599b78469fbd42 not found: ID does not exist"
Sep 11 17:34:08.289986 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.289947    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-workload-certs\") pod \"fc212900-6016-4c2e-af48-5cdcad333b3e\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") "
Sep 11 17:34:08.289986 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.289992    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-token\") pod \"fc212900-6016-4c2e-af48-5cdcad333b3e\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") "
Sep 11 17:34:08.290501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290043    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-credential-socket\") pod \"fc212900-6016-4c2e-af48-5cdcad333b3e\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") "
Sep 11 17:34:08.290501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290079    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-envoy\") pod \"fc212900-6016-4c2e-af48-5cdcad333b3e\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") "
Sep 11 17:34:08.290501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290107    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-workload-socket\") pod \"fc212900-6016-4c2e-af48-5cdcad333b3e\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") "
Sep 11 17:34:08.290501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290148    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-podinfo\") pod \"fc212900-6016-4c2e-af48-5cdcad333b3e\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") "
Sep 11 17:34:08.290501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290222    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-data\") pod \"fc212900-6016-4c2e-af48-5cdcad333b3e\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") "
Sep 11 17:34:08.290501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290259    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc212900-6016-4c2e-af48-5cdcad333b3e-istiod-ca-cert\") pod \"fc212900-6016-4c2e-af48-5cdcad333b3e\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") "
Sep 11 17:34:08.290501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290268    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fc212900-6016-4c2e-af48-5cdcad333b3e" (UID: "fc212900-6016-4c2e-af48-5cdcad333b3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:34:08.290501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290300    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6s78\" (UniqueName: \"kubernetes.io/projected/fc212900-6016-4c2e-af48-5cdcad333b3e-kube-api-access-v6s78\") pod \"fc212900-6016-4c2e-af48-5cdcad333b3e\" (UID: \"fc212900-6016-4c2e-af48-5cdcad333b3e\") "
Sep 11 17:34:08.290501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290363    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fc212900-6016-4c2e-af48-5cdcad333b3e" (UID: "fc212900-6016-4c2e-af48-5cdcad333b3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:34:08.290501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290468    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fc212900-6016-4c2e-af48-5cdcad333b3e" (UID: "fc212900-6016-4c2e-af48-5cdcad333b3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:34:08.291058 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290538    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-data" (OuterVolumeSpecName: "istio-data") pod "fc212900-6016-4c2e-af48-5cdcad333b3e" (UID: "fc212900-6016-4c2e-af48-5cdcad333b3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:34:08.291058 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290688    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc212900-6016-4c2e-af48-5cdcad333b3e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fc212900-6016-4c2e-af48-5cdcad333b3e" (UID: "fc212900-6016-4c2e-af48-5cdcad333b3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:34:08.291058 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290836    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-data\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:34:08.291058 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290857    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc212900-6016-4c2e-af48-5cdcad333b3e-istiod-ca-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:34:08.291058 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290870    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-workload-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:34:08.291058 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290884    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-credential-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:34:08.291058 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.290898    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-workload-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:34:08.292437 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.292411    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-token" (OuterVolumeSpecName: "istio-token") pod "fc212900-6016-4c2e-af48-5cdcad333b3e" (UID: "fc212900-6016-4c2e-af48-5cdcad333b3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:34:08.292574 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.292464    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fc212900-6016-4c2e-af48-5cdcad333b3e" (UID: "fc212900-6016-4c2e-af48-5cdcad333b3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:34:08.292574 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.292500    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fc212900-6016-4c2e-af48-5cdcad333b3e" (UID: "fc212900-6016-4c2e-af48-5cdcad333b3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:34:08.292669 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.292645    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc212900-6016-4c2e-af48-5cdcad333b3e-kube-api-access-v6s78" (OuterVolumeSpecName: "kube-api-access-v6s78") pod "fc212900-6016-4c2e-af48-5cdcad333b3e" (UID: "fc212900-6016-4c2e-af48-5cdcad333b3e"). InnerVolumeSpecName "kube-api-access-v6s78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:34:08.391576 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.391469    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-token\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:34:08.391576 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.391527    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-envoy\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:34:08.391576 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.391535    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc212900-6016-4c2e-af48-5cdcad333b3e-istio-podinfo\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:34:08.391576 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.391543    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6s78\" (UniqueName: \"kubernetes.io/projected/fc212900-6016-4c2e-af48-5cdcad333b3e-kube-api-access-v6s78\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:34:08.525314 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.525283    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"]
Sep 11 17:34:08.530751 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.530721    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b76706e3-gw-openshift-default-7bdcb6d5dd-d6rz7"]
Sep 11 17:34:08.934877 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:08.934840    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:09.036186 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:09.036142    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:09.377125 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:09.377044    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:09.490583 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:09.490545    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:09.579709 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:09.579671    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:10.050950 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:10.050918    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc212900-6016-4c2e-af48-5cdcad333b3e" path="/var/lib/kubelet/pods/fc212900-6016-4c2e-af48-5cdcad333b3e/volumes"
Sep 11 17:34:12.828003 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:12.827966    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:12.828003 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:12.828010    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:12.840839 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:12.840817    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:13.237194 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:13.237163    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:34:17.546331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:17.546289    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:31.978430 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:31.978384    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:32.730822 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:32.730768    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:33.432393 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:33.432356    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:34.240851 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:34.240815    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:34.970729 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:34.970675    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:35.217295 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.217261    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6"]
Sep 11 17:34:35.217667 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.217655    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc212900-6016-4c2e-af48-5cdcad333b3e" containerName="istio-proxy"
Sep 11 17:34:35.217718 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.217669    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc212900-6016-4c2e-af48-5cdcad333b3e" containerName="istio-proxy"
Sep 11 17:34:35.217718 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.217679    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc212900-6016-4c2e-af48-5cdcad333b3e" containerName="istio-proxy"
Sep 11 17:34:35.217718 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.217685    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc212900-6016-4c2e-af48-5cdcad333b3e" containerName="istio-proxy"
Sep 11 17:34:35.217814 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.217757    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc212900-6016-4c2e-af48-5cdcad333b3e" containerName="istio-proxy"
Sep 11 17:34:35.220279 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.220263    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6"
Sep 11 17:34:35.225537 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.225466    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:34:35.225673 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.225545    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:34:35.239774 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.239745    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6"]
Sep 11 17:34:35.354681 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.354645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa8de59e-c542-485b-8679-e53a5686296b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-3005e5-vxdn6\" (UID: \"aa8de59e-c542-485b-8679-e53a5686296b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6"
Sep 11 17:34:35.455667 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.455633    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa8de59e-c542-485b-8679-e53a5686296b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-3005e5-vxdn6\" (UID: \"aa8de59e-c542-485b-8679-e53a5686296b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6"
Sep 11 17:34:35.456276 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.456253    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa8de59e-c542-485b-8679-e53a5686296b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-3005e5-vxdn6\" (UID: \"aa8de59e-c542-485b-8679-e53a5686296b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6"
Sep 11 17:34:35.530508 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.530410    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6"
Sep 11 17:34:35.835904 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.835874    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6"]
Sep 11 17:34:35.838129 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:34:35.838105    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8de59e_c542_485b_8679_e53a5686296b.slice/crio-5b15c33d7f9632602251dc39c44de79c6daefacddb7f793f40a3e2d128b460f9 WatchSource:0}: Error finding container 5b15c33d7f9632602251dc39c44de79c6daefacddb7f793f40a3e2d128b460f9: Status 404 returned error can't find the container with id 5b15c33d7f9632602251dc39c44de79c6daefacddb7f793f40a3e2d128b460f9
Sep 11 17:34:35.902668 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:35.902635    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:36.347876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:36.347835    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6" event={"ID":"aa8de59e-c542-485b-8679-e53a5686296b","Type":"ContainerStarted","Data":"0441dfbf7498babf1743966c598bbe3429161f7a2f0f3eb21952481c0bc377f6"}
Sep 11 17:34:36.348266 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:36.347901    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6" event={"ID":"aa8de59e-c542-485b-8679-e53a5686296b","Type":"ContainerStarted","Data":"5b15c33d7f9632602251dc39c44de79c6daefacddb7f793f40a3e2d128b460f9"}
Sep 11 17:34:36.366981 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:36.366881    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6" podStartSLOduration=1.3668662409999999 podStartE2EDuration="1.366866241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:34:36.364528657 +0000 UTC m=+846.796981434" watchObservedRunningTime="2026-09-11 17:34:36.366866241 +0000 UTC m=+846.799319019"
Sep 11 17:34:36.824577 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:36.824543    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:36.953443 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:36.953387    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj"]
Sep 11 17:34:36.953997 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:36.953977    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc212900-6016-4c2e-af48-5cdcad333b3e" containerName="istio-proxy"
Sep 11 17:34:36.955968 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:36.955950    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj"
Sep 11 17:34:36.966354 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:36.966324    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj"]
Sep 11 17:34:37.073128 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:37.073082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e88fac4b-25e1-4630-ada3-588f6187333b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj\" (UID: \"e88fac4b-25e1-4630-ada3-588f6187333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj"
Sep 11 17:34:37.174713 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:37.174677    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e88fac4b-25e1-4630-ada3-588f6187333b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj\" (UID: \"e88fac4b-25e1-4630-ada3-588f6187333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj"
Sep 11 17:34:37.175386 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:37.175362    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e88fac4b-25e1-4630-ada3-588f6187333b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj\" (UID: \"e88fac4b-25e1-4630-ada3-588f6187333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj"
Sep 11 17:34:37.268229 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:37.268167    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj"
Sep 11 17:34:37.356885 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:37.356846    2560 generic.go:356] "Generic (PLEG): container finished" podID="aa8de59e-c542-485b-8679-e53a5686296b" containerID="0441dfbf7498babf1743966c598bbe3429161f7a2f0f3eb21952481c0bc377f6" exitCode=0
Sep 11 17:34:37.357350 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:37.356995    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6" event={"ID":"aa8de59e-c542-485b-8679-e53a5686296b","Type":"ContainerDied","Data":"0441dfbf7498babf1743966c598bbe3429161f7a2f0f3eb21952481c0bc377f6"}
Sep 11 17:34:37.544803 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:37.544780    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:37.545682 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:37.545660    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj"]
Sep 11 17:34:37.547336 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:34:37.547309    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88fac4b_25e1_4630_ada3_588f6187333b.slice/crio-6baf8cb202e3f3ed91a29827d0c886e7b11064e8ef5154f726e1252d9b353517 WatchSource:0}: Error finding container 6baf8cb202e3f3ed91a29827d0c886e7b11064e8ef5154f726e1252d9b353517: Status 404 returned error can't find the container with id 6baf8cb202e3f3ed91a29827d0c886e7b11064e8ef5154f726e1252d9b353517
Sep 11 17:34:38.224030 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:38.223995    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:38.363345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:38.363311    2560 generic.go:356] "Generic (PLEG): container finished" podID="e88fac4b-25e1-4630-ada3-588f6187333b" containerID="a26e38c5b5dece80e15f97ad8dabd7dd68e7248a377894eb23fe2a6cd1aa5818" exitCode=0
Sep 11 17:34:38.363818 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:38.363391    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj" event={"ID":"e88fac4b-25e1-4630-ada3-588f6187333b","Type":"ContainerDied","Data":"a26e38c5b5dece80e15f97ad8dabd7dd68e7248a377894eb23fe2a6cd1aa5818"}
Sep 11 17:34:38.363818 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:38.363440    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj" event={"ID":"e88fac4b-25e1-4630-ada3-588f6187333b","Type":"ContainerStarted","Data":"6baf8cb202e3f3ed91a29827d0c886e7b11064e8ef5154f726e1252d9b353517"}
Sep 11 17:34:38.499802 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:38.499773    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6"
Sep 11 17:34:38.588599 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:38.588561    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa8de59e-c542-485b-8679-e53a5686296b-maas-api-service-ca\") pod \"aa8de59e-c542-485b-8679-e53a5686296b\" (UID: \"aa8de59e-c542-485b-8679-e53a5686296b\") "
Sep 11 17:34:38.588927 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:38.588906    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa8de59e-c542-485b-8679-e53a5686296b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa8de59e-c542-485b-8679-e53a5686296b" (UID: "aa8de59e-c542-485b-8679-e53a5686296b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:34:38.689785 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:38.689751    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa8de59e-c542-485b-8679-e53a5686296b-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:34:39.029865 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:39.029835    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:39.370199 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:39.370113    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6"
Sep 11 17:34:39.370199 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:39.370145    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6" event={"ID":"aa8de59e-c542-485b-8679-e53a5686296b","Type":"ContainerDied","Data":"5b15c33d7f9632602251dc39c44de79c6daefacddb7f793f40a3e2d128b460f9"}
Sep 11 17:34:39.370199 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:39.370191    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b15c33d7f9632602251dc39c44de79c6daefacddb7f793f40a3e2d128b460f9"
Sep 11 17:34:39.510200 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:39.510173    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj"
Sep 11 17:34:39.600117 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:39.600080    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e88fac4b-25e1-4630-ada3-588f6187333b-maas-api-service-ca\") pod \"e88fac4b-25e1-4630-ada3-588f6187333b\" (UID: \"e88fac4b-25e1-4630-ada3-588f6187333b\") "
Sep 11 17:34:39.600504 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:39.600454    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88fac4b-25e1-4630-ada3-588f6187333b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e88fac4b-25e1-4630-ada3-588f6187333b" (UID: "e88fac4b-25e1-4630-ada3-588f6187333b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:34:39.642530 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:39.642498    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:39.701238 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:39.701201    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e88fac4b-25e1-4630-ada3-588f6187333b-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:34:39.724671 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:39.724637    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:40.376864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:40.376778    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj"
Sep 11 17:34:40.376864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:40.376790    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj" event={"ID":"e88fac4b-25e1-4630-ada3-588f6187333b","Type":"ContainerDied","Data":"6baf8cb202e3f3ed91a29827d0c886e7b11064e8ef5154f726e1252d9b353517"}
Sep 11 17:34:40.376864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:40.376863    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6baf8cb202e3f3ed91a29827d0c886e7b11064e8ef5154f726e1252d9b353517"
Sep 11 17:34:40.523518 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:40.523487    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:42.926417 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:42.926378    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"]
Sep 11 17:34:42.926990 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:42.926970    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa8de59e-c542-485b-8679-e53a5686296b" containerName="revoke-keys"
Sep 11 17:34:42.927065 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:42.926993    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa8de59e-c542-485b-8679-e53a5686296b" containerName="revoke-keys"
Sep 11 17:34:42.927065 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:42.927005    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e88fac4b-25e1-4630-ada3-588f6187333b" containerName="revoke-keys"
Sep 11 17:34:42.927065 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:42.927014    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e88fac4b-25e1-4630-ada3-588f6187333b" containerName="revoke-keys"
Sep 11 17:34:42.927214 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:42.927133    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e88fac4b-25e1-4630-ada3-588f6187333b" containerName="revoke-keys"
Sep 11 17:34:42.927214 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:42.927150    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa8de59e-c542-485b-8679-e53a5686296b" containerName="revoke-keys"
Sep 11 17:34:42.929290 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:42.929269    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:42.931870 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:42.931846    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d92fb3\"/\"openshift-service-ca.crt\""
Sep 11 17:34:42.933024 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:42.933002    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d92fb3\"/\"kube-root-ca.crt\""
Sep 11 17:34:42.933129 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:42.933046    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d92fb3\"/\"test-model-d92fb3-kserve-self-signed-certs\""
Sep 11 17:34:42.933129 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:42.933046    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d92fb3\"/\"default-dockercfg-tghrq\""
Sep 11 17:34:42.939245 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:42.939223    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"]
Sep 11 17:34:42.957121 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:42.957088    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:43.032949 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.032914    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0795e847-5aad-4747-a6d2-bcd0bd247a82-tls-certs\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.033117 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.032965    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-dshm\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.033117 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.033097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-tmp-dir\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.033222 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.033132    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd6lw\" (UniqueName: \"kubernetes.io/projected/0795e847-5aad-4747-a6d2-bcd0bd247a82-kube-api-access-dd6lw\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.033222 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.033154    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-model-cache\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.033222 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.033189    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-home\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.134637 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.134594    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-tmp-dir\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.134849 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.134644    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6lw\" (UniqueName: \"kubernetes.io/projected/0795e847-5aad-4747-a6d2-bcd0bd247a82-kube-api-access-dd6lw\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.134849 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.134689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-model-cache\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.134849 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.134720    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-home\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.134849 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.134789    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0795e847-5aad-4747-a6d2-bcd0bd247a82-tls-certs\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.134849 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.134834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-dshm\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.135101 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.135085    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-tmp-dir\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.135166 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.135102    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-model-cache\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.135213 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.135164    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-home\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.137087 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.137064    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-dshm\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.137322 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.137306    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0795e847-5aad-4747-a6d2-bcd0bd247a82-tls-certs\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.143753 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.143730    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6lw\" (UniqueName: \"kubernetes.io/projected/0795e847-5aad-4747-a6d2-bcd0bd247a82-kube-api-access-dd6lw\") pod \"test-model-d92fb3-kserve-5f645f8ddf-9pj5t\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") " pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.239744 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.239646    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:43.513682 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:43.513656    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"]
Sep 11 17:34:43.515383 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:34:43.515356    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0795e847_5aad_4747_a6d2_bcd0bd247a82.slice/crio-0872ac1828dd5581551300b847f05b7439824fe1c99fbeb33dff8747b41c9174 WatchSource:0}: Error finding container 0872ac1828dd5581551300b847f05b7439824fe1c99fbeb33dff8747b41c9174: Status 404 returned error can't find the container with id 0872ac1828dd5581551300b847f05b7439824fe1c99fbeb33dff8747b41c9174
Sep 11 17:34:44.399343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:44.399301    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t" event={"ID":"0795e847-5aad-4747-a6d2-bcd0bd247a82","Type":"ContainerStarted","Data":"704478e6941f0d390effee585f5cd85fe219ab10a73611c8f18321299da8b7c4"}
Sep 11 17:34:44.399745 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:44.399353    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t" event={"ID":"0795e847-5aad-4747-a6d2-bcd0bd247a82","Type":"ContainerStarted","Data":"0872ac1828dd5581551300b847f05b7439824fe1c99fbeb33dff8747b41c9174"}
Sep 11 17:34:44.417222 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:44.417178    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t" podStartSLOduration=2.417162533 podStartE2EDuration="2.417162533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:34:44.415142055 +0000 UTC m=+854.847595023" watchObservedRunningTime="2026-09-11 17:34:44.417162533 +0000 UTC m=+854.849615345"
Sep 11 17:34:45.325551 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:45.325510    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:45.877424 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:45.877391    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:45.944273 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:45.944210    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:46.044278 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:46.044244    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:46.069172 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:46.069132    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6"]
Sep 11 17:34:46.071092 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:46.071070    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-3005e5-vxdn6"]
Sep 11 17:34:46.144964 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:46.144928    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:46.411213 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:46.411131    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:46.511287 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:46.511252    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:47.258624 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:47.258588    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:48.027243 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:48.027204    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:48.052017 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:48.051982    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa8de59e-c542-485b-8679-e53a5686296b" path="/var/lib/kubelet/pods/aa8de59e-c542-485b-8679-e53a5686296b/volumes"
Sep 11 17:34:48.115143 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:48.115109    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:48.182469 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:48.182414    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:48.286210 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:48.286106    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj"]
Sep 11 17:34:48.286888 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:48.286853    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:48.289110 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:48.289090    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d8713f-mk4lj"]
Sep 11 17:34:48.383241 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:48.383207    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:49.069722 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:49.069690    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:49.382997 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:49.382971    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:49.787214 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:49.787124    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:49.890541 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:49.890507    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:50.052775 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:50.052702    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e88fac4b-25e1-4630-ada3-588f6187333b" path="/var/lib/kubelet/pods/e88fac4b-25e1-4630-ada3-588f6187333b/volumes"
Sep 11 17:34:50.130102 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:50.130038    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:50.311686 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:50.311603    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:50.755190 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:50.755158    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:51.388793 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:51.388755    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"]
Sep 11 17:34:51.437998 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:51.437970    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:52.235629 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:52.235593    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:52.472490 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:52.472448    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:52.595942 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:52.595858    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:53.231275 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:53.231245    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:53.240342 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:53.240313    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:53.240697 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:53.240355    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:53.252959 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:53.252936    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:53.457085 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:53.457055    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:34:54.026941 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:54.026914    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:54.087332 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:54.087301    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:54.483798 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:54.483761    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:54.582956 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:54.582922    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:34:54.818450 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:34:54.818361    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:05.531359 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:05.531303    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:06.408205 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.408147    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn" podUID="8fb46243-07f6-406f-ab89-ecb138f0a1e7" containerName="main" containerID="cri-o://b5db9b7f70bc91635182b2ab4b8e47748b0b57faf32deb3445c77b44f07bf571" gracePeriod=45
Sep 11 17:35:06.656590 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.656566    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:35:06.772229 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.772140    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-home\") pod \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") "
Sep 11 17:35:06.772229 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.772204    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-tmp-dir\") pod \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") "
Sep 11 17:35:06.772229 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.772225    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-dshm\") pod \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") "
Sep 11 17:35:06.772510 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.772260    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-model-cache\") pod \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") "
Sep 11 17:35:06.772510 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.772324    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw8dc\" (UniqueName: \"kubernetes.io/projected/8fb46243-07f6-406f-ab89-ecb138f0a1e7-kube-api-access-bw8dc\") pod \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") "
Sep 11 17:35:06.772510 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.772350    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb46243-07f6-406f-ab89-ecb138f0a1e7-tls-certs\") pod \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\" (UID: \"8fb46243-07f6-406f-ab89-ecb138f0a1e7\") "
Sep 11 17:35:06.772695 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.772512    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fb46243-07f6-406f-ab89-ecb138f0a1e7" (UID: "8fb46243-07f6-406f-ab89-ecb138f0a1e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:35:06.772695 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.772545    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fb46243-07f6-406f-ab89-ecb138f0a1e7" (UID: "8fb46243-07f6-406f-ab89-ecb138f0a1e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:35:06.772695 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.772521    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-home" (OuterVolumeSpecName: "home") pod "8fb46243-07f6-406f-ab89-ecb138f0a1e7" (UID: "8fb46243-07f6-406f-ab89-ecb138f0a1e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:35:06.772892 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.772867    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:06.772892 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.772889    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:06.773088 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.772901    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:06.774510 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.774469    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-dshm" (OuterVolumeSpecName: "dshm") pod "8fb46243-07f6-406f-ab89-ecb138f0a1e7" (UID: "8fb46243-07f6-406f-ab89-ecb138f0a1e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:35:06.774510 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.774503    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb46243-07f6-406f-ab89-ecb138f0a1e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fb46243-07f6-406f-ab89-ecb138f0a1e7" (UID: "8fb46243-07f6-406f-ab89-ecb138f0a1e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:35:06.774649 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.774576    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb46243-07f6-406f-ab89-ecb138f0a1e7-kube-api-access-bw8dc" (OuterVolumeSpecName: "kube-api-access-bw8dc") pod "8fb46243-07f6-406f-ab89-ecb138f0a1e7" (UID: "8fb46243-07f6-406f-ab89-ecb138f0a1e7"). InnerVolumeSpecName "kube-api-access-bw8dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:35:06.873680 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.873644    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fb46243-07f6-406f-ab89-ecb138f0a1e7-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:06.873680 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.873676    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw8dc\" (UniqueName: \"kubernetes.io/projected/8fb46243-07f6-406f-ab89-ecb138f0a1e7-kube-api-access-bw8dc\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:06.873680 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:06.873690    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fb46243-07f6-406f-ab89-ecb138f0a1e7-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:07.518391 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:07.518290    2560 generic.go:356] "Generic (PLEG): container finished" podID="8fb46243-07f6-406f-ab89-ecb138f0a1e7" containerID="b5db9b7f70bc91635182b2ab4b8e47748b0b57faf32deb3445c77b44f07bf571" exitCode=0
Sep 11 17:35:07.518391 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:07.518337    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"
Sep 11 17:35:07.518653 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:07.518383    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn" event={"ID":"8fb46243-07f6-406f-ab89-ecb138f0a1e7","Type":"ContainerDied","Data":"b5db9b7f70bc91635182b2ab4b8e47748b0b57faf32deb3445c77b44f07bf571"}
Sep 11 17:35:07.518653 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:07.518429    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn" event={"ID":"8fb46243-07f6-406f-ab89-ecb138f0a1e7","Type":"ContainerDied","Data":"fb8308561ab7c34102a28e61ec1373565ab07552bb0f7be4fe27f506af83d354"}
Sep 11 17:35:07.518653 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:07.518448    2560 scope.go:117] "RemoveContainer" containerID="b5db9b7f70bc91635182b2ab4b8e47748b0b57faf32deb3445c77b44f07bf571"
Sep 11 17:35:07.519997 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:07.519973    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:07.531380 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:07.531364    2560 scope.go:117] "RemoveContainer" containerID="b5db9b7f70bc91635182b2ab4b8e47748b0b57faf32deb3445c77b44f07bf571"
Sep 11 17:35:07.531778 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:35:07.531749    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5db9b7f70bc91635182b2ab4b8e47748b0b57faf32deb3445c77b44f07bf571\": container with ID starting with b5db9b7f70bc91635182b2ab4b8e47748b0b57faf32deb3445c77b44f07bf571 not found: ID does not exist" containerID="b5db9b7f70bc91635182b2ab4b8e47748b0b57faf32deb3445c77b44f07bf571"
Sep 11 17:35:07.531863 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:07.531789    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5db9b7f70bc91635182b2ab4b8e47748b0b57faf32deb3445c77b44f07bf571"} err="failed to get container status \"b5db9b7f70bc91635182b2ab4b8e47748b0b57faf32deb3445c77b44f07bf571\": rpc error: code = NotFound desc = could not find container \"b5db9b7f70bc91635182b2ab4b8e47748b0b57faf32deb3445c77b44f07bf571\": container with ID starting with b5db9b7f70bc91635182b2ab4b8e47748b0b57faf32deb3445c77b44f07bf571 not found: ID does not exist"
Sep 11 17:35:07.541741 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:07.541717    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"]
Sep 11 17:35:07.545571 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:07.545546    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-3005e5/autores-model-3005e5-kserve-578cd9d4c9-pz5gn"]
Sep 11 17:35:08.051790 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:08.051759    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fb46243-07f6-406f-ab89-ecb138f0a1e7" path="/var/lib/kubelet/pods/8fb46243-07f6-406f-ab89-ecb138f0a1e7/volumes"
Sep 11 17:35:15.017032 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:15.016999    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:16.227746 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:16.227714    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:16.925215 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:16.925182    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:17.298106 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:17.298029    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj"]
Sep 11 17:35:17.298445 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:17.298414    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fb46243-07f6-406f-ab89-ecb138f0a1e7" containerName="main"
Sep 11 17:35:17.298445 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:17.298426    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fb46243-07f6-406f-ab89-ecb138f0a1e7" containerName="main"
Sep 11 17:35:17.298562 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:17.298537    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fb46243-07f6-406f-ab89-ecb138f0a1e7" containerName="main"
Sep 11 17:35:17.302051 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:17.302035    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj"
Sep 11 17:35:17.304613 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:17.304590    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:35:17.304749 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:17.304620    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:35:17.309595 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:17.309573    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj"]
Sep 11 17:35:17.482324 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:17.482293    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96249bc3-5c83-48f0-9250-4206811d878e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj\" (UID: \"96249bc3-5c83-48f0-9250-4206811d878e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj"
Sep 11 17:35:17.583311 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:17.583233    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96249bc3-5c83-48f0-9250-4206811d878e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj\" (UID: \"96249bc3-5c83-48f0-9250-4206811d878e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj"
Sep 11 17:35:17.583851 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:17.583830    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96249bc3-5c83-48f0-9250-4206811d878e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj\" (UID: \"96249bc3-5c83-48f0-9250-4206811d878e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj"
Sep 11 17:35:17.641360 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:17.641326    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj"
Sep 11 17:35:17.708027 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:17.707999    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:17.913253 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:17.913226    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj"]
Sep 11 17:35:17.915607 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:35:17.915580    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96249bc3_5c83_48f0_9250_4206811d878e.slice/crio-67fb58b33f394e1ad751e6176df5c09e7e685846d3d30c8443508be28a23adb9 WatchSource:0}: Error finding container 67fb58b33f394e1ad751e6176df5c09e7e685846d3d30c8443508be28a23adb9: Status 404 returned error can't find the container with id 67fb58b33f394e1ad751e6176df5c09e7e685846d3d30c8443508be28a23adb9
Sep 11 17:35:18.526577 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:18.526492    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:18.574973 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:18.574939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj" event={"ID":"96249bc3-5c83-48f0-9250-4206811d878e","Type":"ContainerStarted","Data":"bee0d5a1b54be99fe60624dd781d1979d060fcc5e71dd8051389cb163155a6bb"}
Sep 11 17:35:18.575139 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:18.574995    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj" event={"ID":"96249bc3-5c83-48f0-9250-4206811d878e","Type":"ContainerStarted","Data":"67fb58b33f394e1ad751e6176df5c09e7e685846d3d30c8443508be28a23adb9"}
Sep 11 17:35:19.341070 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:19.341037    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:19.581277 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:19.581246    2560 generic.go:356] "Generic (PLEG): container finished" podID="96249bc3-5c83-48f0-9250-4206811d878e" containerID="bee0d5a1b54be99fe60624dd781d1979d060fcc5e71dd8051389cb163155a6bb" exitCode=0
Sep 11 17:35:19.581665 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:19.581336    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj" event={"ID":"96249bc3-5c83-48f0-9250-4206811d878e","Type":"ContainerDied","Data":"bee0d5a1b54be99fe60624dd781d1979d060fcc5e71dd8051389cb163155a6bb"}
Sep 11 17:35:20.104762 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:20.104726    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:20.722562 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:20.722535    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj"
Sep 11 17:35:20.817589 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:20.817550    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96249bc3-5c83-48f0-9250-4206811d878e-maas-api-service-ca\") pod \"96249bc3-5c83-48f0-9250-4206811d878e\" (UID: \"96249bc3-5c83-48f0-9250-4206811d878e\") "
Sep 11 17:35:20.817920 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:20.817899    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96249bc3-5c83-48f0-9250-4206811d878e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96249bc3-5c83-48f0-9250-4206811d878e" (UID: "96249bc3-5c83-48f0-9250-4206811d878e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:35:20.918335 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:20.918304    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96249bc3-5c83-48f0-9250-4206811d878e-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:21.592729 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:21.592698    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj" event={"ID":"96249bc3-5c83-48f0-9250-4206811d878e","Type":"ContainerDied","Data":"67fb58b33f394e1ad751e6176df5c09e7e685846d3d30c8443508be28a23adb9"}
Sep 11 17:35:21.592729 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:21.592730    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj"
Sep 11 17:35:21.592933 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:21.592744    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67fb58b33f394e1ad751e6176df5c09e7e685846d3d30c8443508be28a23adb9"
Sep 11 17:35:23.939828 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:23.939735    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:24.515769 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:24.515738    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:24.585837 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:24.585807    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:24.685329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:24.685290    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:24.717980 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:24.717950    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj"]
Sep 11 17:35:24.721190 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:24.721159    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c4a3e530-jsrgj"]
Sep 11 17:35:24.760213 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:24.760182    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:24.903132 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:24.903065    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:25.276727 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:25.276589    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:25.570628 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:25.570521    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:25.662524 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:25.662465    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:26.050989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:26.050957    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96249bc3-5c83-48f0-9250-4206811d878e" path="/var/lib/kubelet/pods/96249bc3-5c83-48f0-9250-4206811d878e/volumes"
Sep 11 17:35:26.182860 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:26.182804    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:27.043091 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:27.043061    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:27.141715 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:27.141685    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:27.813901 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:27.813872    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:28.691753 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:28.691719    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:30.029052 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:30.029018    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:35:30.030361 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:30.030334    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:35:32.992025 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:32.991990    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:33.097465 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:33.097429    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:33.194572 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:33.194542    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:33.289977 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:33.289887    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:33.685696 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:33.685663    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:35.614337 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.614306    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"]
Sep 11 17:35:35.614779 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.614762    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96249bc3-5c83-48f0-9250-4206811d878e" containerName="revoke-keys"
Sep 11 17:35:35.614779 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.614780    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="96249bc3-5c83-48f0-9250-4206811d878e" containerName="revoke-keys"
Sep 11 17:35:35.614890 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.614852    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="96249bc3-5c83-48f0-9250-4206811d878e" containerName="revoke-keys"
Sep 11 17:35:35.619351 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.619331    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.621761 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.621740    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-399fe55a-kserve-self-signed-certs\""
Sep 11 17:35:35.630016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.629984    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"]
Sep 11 17:35:35.650769 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.650739    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:35.657980 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.657959    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56d96\" (UniqueName: \"kubernetes.io/projected/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-kube-api-access-56d96\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.658065 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.657991    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-home\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.658065 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.658019    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-dshm\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.658174 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.658133    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.658224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.658179    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-model-cache\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.658271 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.658230    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tmp-dir\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.758956 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.758924    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56d96\" (UniqueName: \"kubernetes.io/projected/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-kube-api-access-56d96\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.758956 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.758958    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-home\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.759225 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.758986    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-dshm\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.759225 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.759034    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.759225 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.759057    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-model-cache\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.759225 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.759088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tmp-dir\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.759400 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.759367    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-home\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.759523 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.759494    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-model-cache\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.759643 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.759549    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tmp-dir\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.761294 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.761279    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-dshm\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.761601 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.761582    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.766689 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.766663    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56d96\" (UniqueName: \"kubernetes.io/projected/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-kube-api-access-56d96\") pod \"e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") " pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:35.928688 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:35.928646    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:36.202800 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:36.202772    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"]
Sep 11 17:35:36.204653 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:35:36.204621    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5cacf7d_bad8_41bf_8c6a_1cc9f57ccbf6.slice/crio-fde54617f7879e3f153b2c65f545ed75ccfac6b2480d2202ea11f6d12043626d WatchSource:0}: Error finding container fde54617f7879e3f153b2c65f545ed75ccfac6b2480d2202ea11f6d12043626d: Status 404 returned error can't find the container with id fde54617f7879e3f153b2c65f545ed75ccfac6b2480d2202ea11f6d12043626d
Sep 11 17:35:36.673917 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:36.673885    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml" event={"ID":"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6","Type":"ContainerStarted","Data":"257afae7c642dbe69ebf946f9ca63e824550e69d67ed41d7fb773786e2387c91"}
Sep 11 17:35:36.674304 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:36.673925    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml" event={"ID":"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6","Type":"ContainerStarted","Data":"fde54617f7879e3f153b2c65f545ed75ccfac6b2480d2202ea11f6d12043626d"}
Sep 11 17:35:36.692888 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:36.692835    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml" podStartSLOduration=1.6928196340000001 podStartE2EDuration="1.692819634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:35:36.691416855 +0000 UTC m=+907.123869634" watchObservedRunningTime="2026-09-11 17:35:36.692819634 +0000 UTC m=+907.125272412"
Sep 11 17:35:43.134598 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:43.134556    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:43.235024 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:43.234992    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:43.369964 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:43.369930    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"]
Sep 11 17:35:43.370173 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:43.370137    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s" podUID="0e169b63-7546-48ce-85e3-5b807c0dc987" containerName="istio-proxy" containerID="cri-o://a10d89d94a8c9ba6c0ea85a7fffc4445f52092a68bf884be72b986c71c9e9d3f" gracePeriod=30
Sep 11 17:35:43.389560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:43.389415    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:43.805220 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:43.805119    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:44.117235 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:44.117148    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:44.437589 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:44.437546    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:45.928914 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:45.928877    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:45.929353 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:45.929134    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:45.941829 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:45.941803    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:46.739981 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:46.739953    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:35:47.170353 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:47.170313    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:47.381441 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:47.381384    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:47.475505 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:47.475373    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:48.098887 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.098810    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:48.517333 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.517303    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"]
Sep 11 17:35:48.541771 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.541728    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:48.640124 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.640097    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:35:48.739813 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.739781    2560 generic.go:356] "Generic (PLEG): container finished" podID="0e169b63-7546-48ce-85e3-5b807c0dc987" containerID="a10d89d94a8c9ba6c0ea85a7fffc4445f52092a68bf884be72b986c71c9e9d3f" exitCode=0
Sep 11 17:35:48.740017 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.739843    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"
Sep 11 17:35:48.740017 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.739863    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s" event={"ID":"0e169b63-7546-48ce-85e3-5b807c0dc987","Type":"ContainerDied","Data":"a10d89d94a8c9ba6c0ea85a7fffc4445f52092a68bf884be72b986c71c9e9d3f"}
Sep 11 17:35:48.740017 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.739913    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s" event={"ID":"0e169b63-7546-48ce-85e3-5b807c0dc987","Type":"ContainerDied","Data":"3dc58c59e275a6b66dad33a51715ab08a5d33620523860cde08e61e28e6c5a1a"}
Sep 11 17:35:48.740017 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.739937    2560 scope.go:117] "RemoveContainer" containerID="a10d89d94a8c9ba6c0ea85a7fffc4445f52092a68bf884be72b986c71c9e9d3f"
Sep 11 17:35:48.749352 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.749330    2560 scope.go:117] "RemoveContainer" containerID="cf16fec358971e100113acb5fe5d156432e8b122c03d0f191e304cf620dfd89c"
Sep 11 17:35:48.757351 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.757335    2560 scope.go:117] "RemoveContainer" containerID="a10d89d94a8c9ba6c0ea85a7fffc4445f52092a68bf884be72b986c71c9e9d3f"
Sep 11 17:35:48.757635 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:35:48.757617    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10d89d94a8c9ba6c0ea85a7fffc4445f52092a68bf884be72b986c71c9e9d3f\": container with ID starting with a10d89d94a8c9ba6c0ea85a7fffc4445f52092a68bf884be72b986c71c9e9d3f not found: ID does not exist" containerID="a10d89d94a8c9ba6c0ea85a7fffc4445f52092a68bf884be72b986c71c9e9d3f"
Sep 11 17:35:48.757703 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.757642    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10d89d94a8c9ba6c0ea85a7fffc4445f52092a68bf884be72b986c71c9e9d3f"} err="failed to get container status \"a10d89d94a8c9ba6c0ea85a7fffc4445f52092a68bf884be72b986c71c9e9d3f\": rpc error: code = NotFound desc = could not find container \"a10d89d94a8c9ba6c0ea85a7fffc4445f52092a68bf884be72b986c71c9e9d3f\": container with ID starting with a10d89d94a8c9ba6c0ea85a7fffc4445f52092a68bf884be72b986c71c9e9d3f not found: ID does not exist"
Sep 11 17:35:48.757703 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.757657    2560 scope.go:117] "RemoveContainer" containerID="cf16fec358971e100113acb5fe5d156432e8b122c03d0f191e304cf620dfd89c"
Sep 11 17:35:48.757920 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:35:48.757902    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf16fec358971e100113acb5fe5d156432e8b122c03d0f191e304cf620dfd89c\": container with ID starting with cf16fec358971e100113acb5fe5d156432e8b122c03d0f191e304cf620dfd89c not found: ID does not exist" containerID="cf16fec358971e100113acb5fe5d156432e8b122c03d0f191e304cf620dfd89c"
Sep 11 17:35:48.757975 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.757923    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf16fec358971e100113acb5fe5d156432e8b122c03d0f191e304cf620dfd89c"} err="failed to get container status \"cf16fec358971e100113acb5fe5d156432e8b122c03d0f191e304cf620dfd89c\": rpc error: code = NotFound desc = could not find container \"cf16fec358971e100113acb5fe5d156432e8b122c03d0f191e304cf620dfd89c\": container with ID starting with cf16fec358971e100113acb5fe5d156432e8b122c03d0f191e304cf620dfd89c not found: ID does not exist"
Sep 11 17:35:48.792204 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.792172    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-workload-certs\") pod \"0e169b63-7546-48ce-85e3-5b807c0dc987\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") "
Sep 11 17:35:48.792365 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.792222    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-token\") pod \"0e169b63-7546-48ce-85e3-5b807c0dc987\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") "
Sep 11 17:35:48.792365 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.792282    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-897d9\" (UniqueName: \"kubernetes.io/projected/0e169b63-7546-48ce-85e3-5b807c0dc987-kube-api-access-897d9\") pod \"0e169b63-7546-48ce-85e3-5b807c0dc987\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") "
Sep 11 17:35:48.792365 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.792333    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-credential-socket\") pod \"0e169b63-7546-48ce-85e3-5b807c0dc987\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") "
Sep 11 17:35:48.792573 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.792379    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-envoy\") pod \"0e169b63-7546-48ce-85e3-5b807c0dc987\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") "
Sep 11 17:35:48.792573 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.792407    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e169b63-7546-48ce-85e3-5b807c0dc987-istiod-ca-cert\") pod \"0e169b63-7546-48ce-85e3-5b807c0dc987\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") "
Sep 11 17:35:48.792573 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.792466    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-data\") pod \"0e169b63-7546-48ce-85e3-5b807c0dc987\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") "
Sep 11 17:35:48.792573 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.792461    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0e169b63-7546-48ce-85e3-5b807c0dc987" (UID: "0e169b63-7546-48ce-85e3-5b807c0dc987"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:35:48.792782 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.792714    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-podinfo\") pod \"0e169b63-7546-48ce-85e3-5b807c0dc987\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") "
Sep 11 17:35:48.792782 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.792710    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0e169b63-7546-48ce-85e3-5b807c0dc987" (UID: "0e169b63-7546-48ce-85e3-5b807c0dc987"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:35:48.792782 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.792760    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-workload-socket\") pod \"0e169b63-7546-48ce-85e3-5b807c0dc987\" (UID: \"0e169b63-7546-48ce-85e3-5b807c0dc987\") "
Sep 11 17:35:48.793102 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.793072    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e169b63-7546-48ce-85e3-5b807c0dc987-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0e169b63-7546-48ce-85e3-5b807c0dc987" (UID: "0e169b63-7546-48ce-85e3-5b807c0dc987"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:35:48.793209 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.793196    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-workload-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:48.793291 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.793214    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-credential-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:48.793291 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.793228    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e169b63-7546-48ce-85e3-5b807c0dc987-istiod-ca-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:48.793430 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.793287    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-data" (OuterVolumeSpecName: "istio-data") pod "0e169b63-7546-48ce-85e3-5b807c0dc987" (UID: "0e169b63-7546-48ce-85e3-5b807c0dc987"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:35:48.793430 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.793347    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0e169b63-7546-48ce-85e3-5b807c0dc987" (UID: "0e169b63-7546-48ce-85e3-5b807c0dc987"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:35:48.794799 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.794771    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0e169b63-7546-48ce-85e3-5b807c0dc987" (UID: "0e169b63-7546-48ce-85e3-5b807c0dc987"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:35:48.794922 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.794830    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e169b63-7546-48ce-85e3-5b807c0dc987" (UID: "0e169b63-7546-48ce-85e3-5b807c0dc987"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:35:48.795085 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.795065    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0e169b63-7546-48ce-85e3-5b807c0dc987" (UID: "0e169b63-7546-48ce-85e3-5b807c0dc987"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:35:48.795150 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.795112    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e169b63-7546-48ce-85e3-5b807c0dc987-kube-api-access-897d9" (OuterVolumeSpecName: "kube-api-access-897d9") pod "0e169b63-7546-48ce-85e3-5b807c0dc987" (UID: "0e169b63-7546-48ce-85e3-5b807c0dc987"). InnerVolumeSpecName "kube-api-access-897d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:35:48.894054 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.893971    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-envoy\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:48.894054 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.893996    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-data\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:48.894054 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.894006    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-podinfo\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:48.894054 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.894015    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e169b63-7546-48ce-85e3-5b807c0dc987-workload-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:48.894054 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.894023    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e169b63-7546-48ce-85e3-5b807c0dc987-istio-token\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:48.894054 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:48.894031    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-897d9\" (UniqueName: \"kubernetes.io/projected/0e169b63-7546-48ce-85e3-5b807c0dc987-kube-api-access-897d9\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:35:48.894511 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:35:48.894125    2560 secret.go:189] Couldn't get secret llm/e2e-conflict-b-399fe55a-kserve-self-signed-certs: secret "e2e-conflict-b-399fe55a-kserve-self-signed-certs" not found
Sep 11 17:35:48.894511 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:35:48.894212    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs podName:b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6 nodeName:}" failed. No retries permitted until 2026-09-11 17:35:49.394191029 +0000 UTC m=+919.826643795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs") pod "e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml" (UID: "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6") : secret "e2e-conflict-b-399fe55a-kserve-self-signed-certs" not found
Sep 11 17:35:49.071026 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:49.070989    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"]
Sep 11 17:35:49.077517 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:49.077487    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s"]
Sep 11 17:35:49.077719 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:49.077694    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:49.157723 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:49.157628    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:49.398432 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:35:49.398400    2560 secret.go:189] Couldn't get secret llm/e2e-conflict-b-399fe55a-kserve-self-signed-certs: secret "e2e-conflict-b-399fe55a-kserve-self-signed-certs" not found
Sep 11 17:35:49.398618 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:35:49.398462    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs podName:b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6 nodeName:}" failed. No retries permitted until 2026-09-11 17:35:50.398448815 +0000 UTC m=+920.830901571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs") pod "e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml" (UID: "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6") : secret "e2e-conflict-b-399fe55a-kserve-self-signed-certs" not found
Sep 11 17:35:49.465394 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:49.465321    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:49.587680 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:49.587651    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:49.595646 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:49.595609    2560 patch_prober.go:28] interesting pod/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 17:35:49.595798 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:49.595671    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-3005e5-openshift-default-64f9944598-m8m4s" podUID="0e169b63-7546-48ce-85e3-5b807c0dc987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 17:35:49.680908 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:49.680878    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:50.051675 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:50.051646    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e169b63-7546-48ce-85e3-5b807c0dc987" path="/var/lib/kubelet/pods/0e169b63-7546-48ce-85e3-5b807c0dc987/volumes"
Sep 11 17:35:50.407728 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:35:50.407697    2560 secret.go:189] Couldn't get secret llm/e2e-conflict-b-399fe55a-kserve-self-signed-certs: secret "e2e-conflict-b-399fe55a-kserve-self-signed-certs" not found
Sep 11 17:35:50.407896 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:35:50.407775    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs podName:b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6 nodeName:}" failed. No retries permitted until 2026-09-11 17:35:52.407759177 +0000 UTC m=+922.840211946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs") pod "e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml" (UID: "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6") : secret "e2e-conflict-b-399fe55a-kserve-self-signed-certs" not found
Sep 11 17:35:51.923105 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:51.923074    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"]
Sep 11 17:35:51.969755 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:51.969721    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:35:52.427912 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:35:52.427876    2560 secret.go:189] Couldn't get secret llm/e2e-conflict-b-399fe55a-kserve-self-signed-certs: secret "e2e-conflict-b-399fe55a-kserve-self-signed-certs" not found
Sep 11 17:35:52.428074 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:35:52.427972    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs podName:b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6 nodeName:}" failed. No retries permitted until 2026-09-11 17:35:56.427952008 +0000 UTC m=+926.860404777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs") pod "e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml" (UID: "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6") : secret "e2e-conflict-b-399fe55a-kserve-self-signed-certs" not found
Sep 11 17:35:56.469015 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:35:56.468983    2560 secret.go:189] Couldn't get secret llm/e2e-conflict-b-399fe55a-kserve-self-signed-certs: secret "e2e-conflict-b-399fe55a-kserve-self-signed-certs" not found
Sep 11 17:35:56.469380 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:35:56.469056    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs podName:b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6 nodeName:}" failed. No retries permitted until 2026-09-11 17:36:04.469041788 +0000 UTC m=+934.901494544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs") pod "e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml" (UID: "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6") : secret "e2e-conflict-b-399fe55a-kserve-self-signed-certs" not found
Sep 11 17:35:59.622004 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:35:59.621965    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:04.550971 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:36:04.550936    2560 secret.go:189] Couldn't get secret llm/e2e-conflict-b-399fe55a-kserve-self-signed-certs: secret "e2e-conflict-b-399fe55a-kserve-self-signed-certs" not found
Sep 11 17:36:04.551343 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:36:04.551012    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs podName:b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6 nodeName:}" failed. No retries permitted until 2026-09-11 17:36:20.550999217 +0000 UTC m=+950.983451972 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs") pod "e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml" (UID: "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6") : secret "e2e-conflict-b-399fe55a-kserve-self-signed-certs" not found
Sep 11 17:36:04.766802 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:04.766760    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml" podUID="b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6" containerName="main" containerID="cri-o://257afae7c642dbe69ebf946f9ca63e824550e69d67ed41d7fb773786e2387c91" gracePeriod=45
Sep 11 17:36:05.017536 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.017511    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:36:05.056311 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.056280    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-dshm\") pod \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") "
Sep 11 17:36:05.056459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.056379    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56d96\" (UniqueName: \"kubernetes.io/projected/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-kube-api-access-56d96\") pod \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") "
Sep 11 17:36:05.056459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.056420    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs\") pod \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") "
Sep 11 17:36:05.056459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.056448    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-model-cache\") pod \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") "
Sep 11 17:36:05.056643 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.056518    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tmp-dir\") pod \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") "
Sep 11 17:36:05.056643 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.056560    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-home\") pod \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\" (UID: \"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6\") "
Sep 11 17:36:05.056789 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.056760    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6" (UID: "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:36:05.056915 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.056796    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6" (UID: "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:36:05.056984 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.056958    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-home" (OuterVolumeSpecName: "home") pod "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6" (UID: "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:36:05.057049 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.057030    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:36:05.057105 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.057055    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:36:05.058617 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.058589    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-dshm" (OuterVolumeSpecName: "dshm") pod "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6" (UID: "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:36:05.058699 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.058664    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-kube-api-access-56d96" (OuterVolumeSpecName: "kube-api-access-56d96") pod "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6" (UID: "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6"). InnerVolumeSpecName "kube-api-access-56d96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:36:05.058882 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.058863    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6" (UID: "b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:36:05.158330 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.158247    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:36:05.158330 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.158276    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56d96\" (UniqueName: \"kubernetes.io/projected/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-kube-api-access-56d96\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:36:05.158330 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.158287    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:36:05.158330 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.158295    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:36:05.325260 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.325228    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:05.395670 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.395637    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:05.475917 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.475844    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:05.574796 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.574751    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:05.829886 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.829793    2560 generic.go:356] "Generic (PLEG): container finished" podID="b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6" containerID="257afae7c642dbe69ebf946f9ca63e824550e69d67ed41d7fb773786e2387c91" exitCode=0
Sep 11 17:36:05.829886 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.829847    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"
Sep 11 17:36:05.829886 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.829881    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml" event={"ID":"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6","Type":"ContainerDied","Data":"257afae7c642dbe69ebf946f9ca63e824550e69d67ed41d7fb773786e2387c91"}
Sep 11 17:36:05.830176 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.829919    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml" event={"ID":"b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6","Type":"ContainerDied","Data":"fde54617f7879e3f153b2c65f545ed75ccfac6b2480d2202ea11f6d12043626d"}
Sep 11 17:36:05.830176 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.829937    2560 scope.go:117] "RemoveContainer" containerID="257afae7c642dbe69ebf946f9ca63e824550e69d67ed41d7fb773786e2387c91"
Sep 11 17:36:05.839500 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.839464    2560 scope.go:117] "RemoveContainer" containerID="257afae7c642dbe69ebf946f9ca63e824550e69d67ed41d7fb773786e2387c91"
Sep 11 17:36:05.839795 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:36:05.839778    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257afae7c642dbe69ebf946f9ca63e824550e69d67ed41d7fb773786e2387c91\": container with ID starting with 257afae7c642dbe69ebf946f9ca63e824550e69d67ed41d7fb773786e2387c91 not found: ID does not exist" containerID="257afae7c642dbe69ebf946f9ca63e824550e69d67ed41d7fb773786e2387c91"
Sep 11 17:36:05.839866 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.839803    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257afae7c642dbe69ebf946f9ca63e824550e69d67ed41d7fb773786e2387c91"} err="failed to get container status \"257afae7c642dbe69ebf946f9ca63e824550e69d67ed41d7fb773786e2387c91\": rpc error: code = NotFound desc = could not find container \"257afae7c642dbe69ebf946f9ca63e824550e69d67ed41d7fb773786e2387c91\": container with ID starting with 257afae7c642dbe69ebf946f9ca63e824550e69d67ed41d7fb773786e2387c91 not found: ID does not exist"
Sep 11 17:36:05.851314 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.851291    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"]
Sep 11 17:36:05.857577 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:05.857555    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-399fe55a-kserve-7fc56c686b-fvqml"]
Sep 11 17:36:06.051105 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:06.051072    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6" path="/var/lib/kubelet/pods/b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6/volumes"
Sep 11 17:36:06.941952 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:06.941914    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858" podUID="b2896948-942e-40cd-a2ed-a01dbe5944af" containerName="main" containerID="cri-o://d0050eea5e5e7ecf3923a4d8a82911a7e5e41f01c8d232d245591c616a039484" gracePeriod=45
Sep 11 17:36:07.198665 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.198608    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:36:07.278284 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.278244    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2896948-942e-40cd-a2ed-a01dbe5944af-tls-certs\") pod \"b2896948-942e-40cd-a2ed-a01dbe5944af\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") "
Sep 11 17:36:07.278433 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.278341    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-home\") pod \"b2896948-942e-40cd-a2ed-a01dbe5944af\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") "
Sep 11 17:36:07.278433 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.278417    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-tmp-dir\") pod \"b2896948-942e-40cd-a2ed-a01dbe5944af\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") "
Sep 11 17:36:07.278571 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.278459    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zr84\" (UniqueName: \"kubernetes.io/projected/b2896948-942e-40cd-a2ed-a01dbe5944af-kube-api-access-4zr84\") pod \"b2896948-942e-40cd-a2ed-a01dbe5944af\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") "
Sep 11 17:36:07.278571 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.278526    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-dshm\") pod \"b2896948-942e-40cd-a2ed-a01dbe5944af\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") "
Sep 11 17:36:07.278666 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.278593    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-model-cache\") pod \"b2896948-942e-40cd-a2ed-a01dbe5944af\" (UID: \"b2896948-942e-40cd-a2ed-a01dbe5944af\") "
Sep 11 17:36:07.278757 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.278712    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-home" (OuterVolumeSpecName: "home") pod "b2896948-942e-40cd-a2ed-a01dbe5944af" (UID: "b2896948-942e-40cd-a2ed-a01dbe5944af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:36:07.278868 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.278772    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2896948-942e-40cd-a2ed-a01dbe5944af" (UID: "b2896948-942e-40cd-a2ed-a01dbe5944af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:36:07.278936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.278916    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2896948-942e-40cd-a2ed-a01dbe5944af" (UID: "b2896948-942e-40cd-a2ed-a01dbe5944af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:36:07.279056 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.279040    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:36:07.279100 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.279062    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:36:07.279100 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.279076    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:36:07.280557 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.280530    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2896948-942e-40cd-a2ed-a01dbe5944af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2896948-942e-40cd-a2ed-a01dbe5944af" (UID: "b2896948-942e-40cd-a2ed-a01dbe5944af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:36:07.280693 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.280675    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-dshm" (OuterVolumeSpecName: "dshm") pod "b2896948-942e-40cd-a2ed-a01dbe5944af" (UID: "b2896948-942e-40cd-a2ed-a01dbe5944af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:36:07.280734 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.280716    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2896948-942e-40cd-a2ed-a01dbe5944af-kube-api-access-4zr84" (OuterVolumeSpecName: "kube-api-access-4zr84") pod "b2896948-942e-40cd-a2ed-a01dbe5944af" (UID: "b2896948-942e-40cd-a2ed-a01dbe5944af"). InnerVolumeSpecName "kube-api-access-4zr84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:36:07.379725 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.379692    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2896948-942e-40cd-a2ed-a01dbe5944af-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:36:07.379725 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.379726    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zr84\" (UniqueName: \"kubernetes.io/projected/b2896948-942e-40cd-a2ed-a01dbe5944af-kube-api-access-4zr84\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:36:07.379911 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.379742    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2896948-942e-40cd-a2ed-a01dbe5944af-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:36:07.842647 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.842611    2560 generic.go:356] "Generic (PLEG): container finished" podID="b2896948-942e-40cd-a2ed-a01dbe5944af" containerID="d0050eea5e5e7ecf3923a4d8a82911a7e5e41f01c8d232d245591c616a039484" exitCode=0
Sep 11 17:36:07.842870 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.842661    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"
Sep 11 17:36:07.842870 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.842668    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858" event={"ID":"b2896948-942e-40cd-a2ed-a01dbe5944af","Type":"ContainerDied","Data":"d0050eea5e5e7ecf3923a4d8a82911a7e5e41f01c8d232d245591c616a039484"}
Sep 11 17:36:07.842870 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.842706    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858" event={"ID":"b2896948-942e-40cd-a2ed-a01dbe5944af","Type":"ContainerDied","Data":"96647070e2a45ff5d22a9f94ce3e8cf0f69973b2daed482f4ca8098df410f797"}
Sep 11 17:36:07.842870 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.842726    2560 scope.go:117] "RemoveContainer" containerID="d0050eea5e5e7ecf3923a4d8a82911a7e5e41f01c8d232d245591c616a039484"
Sep 11 17:36:07.852248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.852230    2560 scope.go:117] "RemoveContainer" containerID="d0050eea5e5e7ecf3923a4d8a82911a7e5e41f01c8d232d245591c616a039484"
Sep 11 17:36:07.852523 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:36:07.852504    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0050eea5e5e7ecf3923a4d8a82911a7e5e41f01c8d232d245591c616a039484\": container with ID starting with d0050eea5e5e7ecf3923a4d8a82911a7e5e41f01c8d232d245591c616a039484 not found: ID does not exist" containerID="d0050eea5e5e7ecf3923a4d8a82911a7e5e41f01c8d232d245591c616a039484"
Sep 11 17:36:07.852576 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.852531    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0050eea5e5e7ecf3923a4d8a82911a7e5e41f01c8d232d245591c616a039484"} err="failed to get container status \"d0050eea5e5e7ecf3923a4d8a82911a7e5e41f01c8d232d245591c616a039484\": rpc error: code = NotFound desc = could not find container \"d0050eea5e5e7ecf3923a4d8a82911a7e5e41f01c8d232d245591c616a039484\": container with ID starting with d0050eea5e5e7ecf3923a4d8a82911a7e5e41f01c8d232d245591c616a039484 not found: ID does not exist"
Sep 11 17:36:07.864849 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.864822    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"]
Sep 11 17:36:07.868775 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:07.868755    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-399fe55a-kserve-65c87cdc76-m5858"]
Sep 11 17:36:08.051153 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:08.051122    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2896948-942e-40cd-a2ed-a01dbe5944af" path="/var/lib/kubelet/pods/b2896948-942e-40cd-a2ed-a01dbe5944af/volumes"
Sep 11 17:36:11.123241 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:11.123205    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:12.681125 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:12.681090    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:12.782888 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:12.782854    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:13.526966 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:13.526932    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:15.247029 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.246977    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"]
Sep 11 17:36:15.247551 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.247534    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e169b63-7546-48ce-85e3-5b807c0dc987" containerName="istio-proxy"
Sep 11 17:36:15.247600 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.247554    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e169b63-7546-48ce-85e3-5b807c0dc987" containerName="istio-proxy"
Sep 11 17:36:15.247600 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.247574    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6" containerName="main"
Sep 11 17:36:15.247600 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.247580    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6" containerName="main"
Sep 11 17:36:15.247600 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.247598    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2896948-942e-40cd-a2ed-a01dbe5944af" containerName="main"
Sep 11 17:36:15.247726 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.247604    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2896948-942e-40cd-a2ed-a01dbe5944af" containerName="main"
Sep 11 17:36:15.247726 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.247672    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e169b63-7546-48ce-85e3-5b807c0dc987" containerName="istio-proxy"
Sep 11 17:36:15.247726 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.247682    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e169b63-7546-48ce-85e3-5b807c0dc987" containerName="istio-proxy"
Sep 11 17:36:15.247726 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.247691    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2896948-942e-40cd-a2ed-a01dbe5944af" containerName="main"
Sep 11 17:36:15.247726 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.247698    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5cacf7d-bad8-41bf-8c6a-1cc9f57ccbf6" containerName="main"
Sep 11 17:36:15.251853 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.251835    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.255638 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.255616    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d148f3\"/\"test-model-d148f3-kserve-self-signed-certs\""
Sep 11 17:36:15.255638 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.255633    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d148f3\"/\"openshift-service-ca.crt\""
Sep 11 17:36:15.255834 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.255616    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d148f3\"/\"default-dockercfg-5gclb\""
Sep 11 17:36:15.255834 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.255616    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d148f3\"/\"kube-root-ca.crt\""
Sep 11 17:36:15.261344 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.261319    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"]
Sep 11 17:36:15.277841 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.277815    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:15.356490 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.356452    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a22fb70-ac37-464c-97bb-96a7b2d44607-tls-certs\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.356658 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.356564    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-model-cache\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.356658 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.356604    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-tmp-dir\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.356732 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.356655    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-home\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.356732 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.356678    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkt24\" (UniqueName: \"kubernetes.io/projected/8a22fb70-ac37-464c-97bb-96a7b2d44607-kube-api-access-lkt24\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.356732 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.356711    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-dshm\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.458119 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.458083    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-model-cache\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.458316 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.458130    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-tmp-dir\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.458316 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.458267    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-home\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.458442 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.458317    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkt24\" (UniqueName: \"kubernetes.io/projected/8a22fb70-ac37-464c-97bb-96a7b2d44607-kube-api-access-lkt24\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.458442 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.458371    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-dshm\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.458568 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.458461    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a22fb70-ac37-464c-97bb-96a7b2d44607-tls-certs\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.458568 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.458544    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-tmp-dir\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.458679 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.458591    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-model-cache\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.458758 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.458740    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-home\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.460804 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.460773    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-dshm\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.461400 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.461379    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a22fb70-ac37-464c-97bb-96a7b2d44607-tls-certs\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.468027 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.467998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkt24\" (UniqueName: \"kubernetes.io/projected/8a22fb70-ac37-464c-97bb-96a7b2d44607-kube-api-access-lkt24\") pod \"test-model-d148f3-kserve-5cb8d65df-x2472\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") " pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.562924 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.562834    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:15.835919 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.835890    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"]
Sep 11 17:36:15.837513 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:36:15.837486    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a22fb70_ac37_464c_97bb_96a7b2d44607.slice/crio-c3b47b8d20c53d0cd9201c9763c1d6c6afc9502916b2a1a09926327f0cc76100 WatchSource:0}: Error finding container c3b47b8d20c53d0cd9201c9763c1d6c6afc9502916b2a1a09926327f0cc76100: Status 404 returned error can't find the container with id c3b47b8d20c53d0cd9201c9763c1d6c6afc9502916b2a1a09926327f0cc76100
Sep 11 17:36:15.887532 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:15.887499    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472" event={"ID":"8a22fb70-ac37-464c-97bb-96a7b2d44607","Type":"ContainerStarted","Data":"c3b47b8d20c53d0cd9201c9763c1d6c6afc9502916b2a1a09926327f0cc76100"}
Sep 11 17:36:16.895253 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:16.895208    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472" event={"ID":"8a22fb70-ac37-464c-97bb-96a7b2d44607","Type":"ContainerStarted","Data":"cf52bc7f08718c78d27ac882de50540bf8c27e42ea46dac789618f31887f0b8a"}
Sep 11 17:36:16.914339 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:16.914292    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472" podStartSLOduration=1.914277008 podStartE2EDuration="1.914277008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:36:16.912644625 +0000 UTC m=+947.345097403" watchObservedRunningTime="2026-09-11 17:36:16.914277008 +0000 UTC m=+947.346729784"
Sep 11 17:36:21.322184 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:21.322147    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:25.563899 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:25.563862    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:25.564283 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:25.563919    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:25.576274 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:25.576248    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:25.953908 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:25.953875    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:36:27.697651 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:27.697610    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:27.761285 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:27.761248    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:27.866953 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:27.866921    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:38.378275 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:38.378242    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:39.139569 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:39.139533    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:45.225674 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:45.225638    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:45.333921 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:45.333888    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:45.425745 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:45.425712    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:45.486436 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:45.486359    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:46.137713 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.137681    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:46.235749 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.235719    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:46.336213 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.336178    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:46.637624 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.637590    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"]
Sep 11 17:36:46.637981 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.637968    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e169b63-7546-48ce-85e3-5b807c0dc987" containerName="istio-proxy"
Sep 11 17:36:46.638030 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.637983    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e169b63-7546-48ce-85e3-5b807c0dc987" containerName="istio-proxy"
Sep 11 17:36:46.642309 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.642292    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.647947 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.647913    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-87b0dc\"/\"openshift-service-ca.crt\""
Sep 11 17:36:46.648154 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.648131    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-87b0dc\"/\"default-dockercfg-8bdv2\""
Sep 11 17:36:46.648318 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.647925    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-87b0dc\"/\"explicit-model-87b0dc-kserve-self-signed-certs\""
Sep 11 17:36:46.648648 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.648038    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-87b0dc\"/\"kube-root-ca.crt\""
Sep 11 17:36:46.656708 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.656686    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"]
Sep 11 17:36:46.681932 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.681907    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:46.751901 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.751872    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-tmp-dir\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.752052 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.751909    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxmcd\" (UniqueName: \"kubernetes.io/projected/4ece20b2-c1d2-4243-ad32-ea6881c22699-kube-api-access-lxmcd\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.752052 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.752011    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ece20b2-c1d2-4243-ad32-ea6881c22699-tls-certs\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.752052 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.752042    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-dshm\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.752161 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.752076    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-home\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.752161 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.752097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-model-cache\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.853147 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.853110    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-tmp-dir\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.853345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.853153    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxmcd\" (UniqueName: \"kubernetes.io/projected/4ece20b2-c1d2-4243-ad32-ea6881c22699-kube-api-access-lxmcd\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.853345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.853223    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ece20b2-c1d2-4243-ad32-ea6881c22699-tls-certs\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.853345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.853247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-dshm\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.853345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.853279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-home\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.853345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.853300    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-model-cache\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.853634 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.853530    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-tmp-dir\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.853695 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.853654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-home\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.853775 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.853755    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-model-cache\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.855490 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.855456    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-dshm\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.855734 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.855716    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ece20b2-c1d2-4243-ad32-ea6881c22699-tls-certs\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.862041 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.862022    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxmcd\" (UniqueName: \"kubernetes.io/projected/4ece20b2-c1d2-4243-ad32-ea6881c22699-kube-api-access-lxmcd\") pod \"explicit-model-87b0dc-kserve-666bf4dfd9-blmnr\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") " pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:46.956548 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:46.956456    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:47.227329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:47.227303    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"]
Sep 11 17:36:47.228815 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:36:47.228788    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ece20b2_c1d2_4243_ad32_ea6881c22699.slice/crio-21dd1da2feeb05f61afd7ea10d5cd11a98fabbd7fd8bd572e80f865682e28d91 WatchSource:0}: Error finding container 21dd1da2feeb05f61afd7ea10d5cd11a98fabbd7fd8bd572e80f865682e28d91: Status 404 returned error can't find the container with id 21dd1da2feeb05f61afd7ea10d5cd11a98fabbd7fd8bd572e80f865682e28d91
Sep 11 17:36:47.800450 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:47.800416    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:48.055589 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:48.055505    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr" event={"ID":"4ece20b2-c1d2-4243-ad32-ea6881c22699","Type":"ContainerStarted","Data":"6ea009223bc1cb45619bae56a617e804356b77dd2e4c01c2d1a6a1b678e9f5ac"}
Sep 11 17:36:48.055589 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:48.055547    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr" event={"ID":"4ece20b2-c1d2-4243-ad32-ea6881c22699","Type":"ContainerStarted","Data":"21dd1da2feeb05f61afd7ea10d5cd11a98fabbd7fd8bd572e80f865682e28d91"}
Sep 11 17:36:48.076030 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:48.075973    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr" podStartSLOduration=2.075958623 podStartE2EDuration="2.075958623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:36:48.073359294 +0000 UTC m=+978.505812084" watchObservedRunningTime="2026-09-11 17:36:48.075958623 +0000 UTC m=+978.508411403"
Sep 11 17:36:53.789651 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:53.789561    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:53.889626 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:53.889597    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:54.459659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:54.459628    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:56.956810 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:56.956767    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:56.957305 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:56.956838    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:56.969610 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:56.969579    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:57.112647 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:57.112611    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:36:59.071785 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:59.071751    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"]
Sep 11 17:36:59.086332 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:59.086302    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:36:59.181171 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:59.181136    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm"]
Sep 11 17:36:59.184826 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:59.184807    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm"
Sep 11 17:36:59.187538 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:59.187515    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:36:59.187538 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:59.187532    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:36:59.192322 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:59.192297    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm"]
Sep 11 17:36:59.281715 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:59.281683    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac3099a-9ad5-4369-a5f0-8394dd851c3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm\" (UID: \"bac3099a-9ad5-4369-a5f0-8394dd851c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm"
Sep 11 17:36:59.383169 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:59.383136    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac3099a-9ad5-4369-a5f0-8394dd851c3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm\" (UID: \"bac3099a-9ad5-4369-a5f0-8394dd851c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm"
Sep 11 17:36:59.383747 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:59.383721    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac3099a-9ad5-4369-a5f0-8394dd851c3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm\" (UID: \"bac3099a-9ad5-4369-a5f0-8394dd851c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm"
Sep 11 17:36:59.498468 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:59.498435    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm"
Sep 11 17:36:59.784024 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:36:59.783995    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm"]
Sep 11 17:36:59.784846 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:36:59.784819    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac3099a_9ad5_4369_a5f0_8394dd851c3b.slice/crio-878796dc8c615d5648af0e1a51d4cfcfe0c5a6669ed10402445c2e2657928ac2 WatchSource:0}: Error finding container 878796dc8c615d5648af0e1a51d4cfcfe0c5a6669ed10402445c2e2657928ac2: Status 404 returned error can't find the container with id 878796dc8c615d5648af0e1a51d4cfcfe0c5a6669ed10402445c2e2657928ac2
Sep 11 17:37:00.111666 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:00.111635    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:00.120263 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:00.120233    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm" event={"ID":"bac3099a-9ad5-4369-a5f0-8394dd851c3b","Type":"ContainerStarted","Data":"878796dc8c615d5648af0e1a51d4cfcfe0c5a6669ed10402445c2e2657928ac2"}
Sep 11 17:37:00.941934 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:00.941904    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:01.127170 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:01.127122    2560 generic.go:356] "Generic (PLEG): container finished" podID="bac3099a-9ad5-4369-a5f0-8394dd851c3b" containerID="577215d4ddeaa818c03e38669656420c26ffd03e96364a7f571b7ccfe4f135fa" exitCode=0
Sep 11 17:37:01.127562 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:01.127211    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm" event={"ID":"bac3099a-9ad5-4369-a5f0-8394dd851c3b","Type":"ContainerDied","Data":"577215d4ddeaa818c03e38669656420c26ffd03e96364a7f571b7ccfe4f135fa"}
Sep 11 17:37:01.398101 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:01.398073    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:01.498053 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:01.498015    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:01.632080 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:01.632053    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:01.733425 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:01.733349    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:02.054438 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:02.054362    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:02.272068 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:02.272046    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm"
Sep 11 17:37:02.412629 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:02.412605    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac3099a-9ad5-4369-a5f0-8394dd851c3b-maas-api-service-ca\") pod \"bac3099a-9ad5-4369-a5f0-8394dd851c3b\" (UID: \"bac3099a-9ad5-4369-a5f0-8394dd851c3b\") "
Sep 11 17:37:02.412944 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:02.412924    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac3099a-9ad5-4369-a5f0-8394dd851c3b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bac3099a-9ad5-4369-a5f0-8394dd851c3b" (UID: "bac3099a-9ad5-4369-a5f0-8394dd851c3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:37:02.512120 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:02.512089    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:02.514306 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:02.514287    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac3099a-9ad5-4369-a5f0-8394dd851c3b-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:03.002649 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:03.002602    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:03.139327 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:03.139303    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm"
Sep 11 17:37:03.139513 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:03.139302    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm" event={"ID":"bac3099a-9ad5-4369-a5f0-8394dd851c3b","Type":"ContainerDied","Data":"878796dc8c615d5648af0e1a51d4cfcfe0c5a6669ed10402445c2e2657928ac2"}
Sep 11 17:37:03.139513 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:03.139421    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="878796dc8c615d5648af0e1a51d4cfcfe0c5a6669ed10402445c2e2657928ac2"
Sep 11 17:37:04.676872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:04.676817    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:04.870163 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:04.870134    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:04.969804 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:04.969714    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:05.754792 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:05.754760    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:06.543119 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:06.543075    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:07.426267 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:07.426241    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:08.128916 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:08.128885    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:08.187428 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:08.187396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:08.552929 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:08.552880    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:08.708577 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:08.708545    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:08.800789 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:08.800739    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:09.470719 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:09.470639    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:09.571452 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:09.571405    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:09.662600 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:09.662563    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm"]
Sep 11 17:37:09.664959 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:09.664915    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:09.665449 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:09.665429    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-87b0dc-rtfsm"]
Sep 11 17:37:09.726246 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:09.726167    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:10.008980 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:10.008900    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:10.051367 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:10.051334    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bac3099a-9ad5-4369-a5f0-8394dd851c3b" path="/var/lib/kubelet/pods/bac3099a-9ad5-4369-a5f0-8394dd851c3b/volumes"
Sep 11 17:37:10.776629 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:10.776595    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:11.514900 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:11.514864    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:12.345994 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:12.345959    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:14.131335 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.131296    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr" podUID="4ece20b2-c1d2-4243-ad32-ea6881c22699" containerName="main" containerID="cri-o://6ea009223bc1cb45619bae56a617e804356b77dd2e4c01c2d1a6a1b678e9f5ac" gracePeriod=45
Sep 11 17:37:14.379943 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.379922    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:37:14.541779 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.541690    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ece20b2-c1d2-4243-ad32-ea6881c22699-tls-certs\") pod \"4ece20b2-c1d2-4243-ad32-ea6881c22699\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") "
Sep 11 17:37:14.541779 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.541749    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxmcd\" (UniqueName: \"kubernetes.io/projected/4ece20b2-c1d2-4243-ad32-ea6881c22699-kube-api-access-lxmcd\") pod \"4ece20b2-c1d2-4243-ad32-ea6881c22699\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") "
Sep 11 17:37:14.541779 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.541778    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-dshm\") pod \"4ece20b2-c1d2-4243-ad32-ea6881c22699\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") "
Sep 11 17:37:14.542102 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.541861    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-tmp-dir\") pod \"4ece20b2-c1d2-4243-ad32-ea6881c22699\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") "
Sep 11 17:37:14.542102 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.541882    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-model-cache\") pod \"4ece20b2-c1d2-4243-ad32-ea6881c22699\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") "
Sep 11 17:37:14.542102 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.541917    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-home\") pod \"4ece20b2-c1d2-4243-ad32-ea6881c22699\" (UID: \"4ece20b2-c1d2-4243-ad32-ea6881c22699\") "
Sep 11 17:37:14.542263 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.542163    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ece20b2-c1d2-4243-ad32-ea6881c22699" (UID: "4ece20b2-c1d2-4243-ad32-ea6881c22699"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:37:14.542263 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.542150    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ece20b2-c1d2-4243-ad32-ea6881c22699" (UID: "4ece20b2-c1d2-4243-ad32-ea6881c22699"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:37:14.542263 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.542203    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-home" (OuterVolumeSpecName: "home") pod "4ece20b2-c1d2-4243-ad32-ea6881c22699" (UID: "4ece20b2-c1d2-4243-ad32-ea6881c22699"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:37:14.542378 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.542333    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:14.542378 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.542354    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:14.542378 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.542368    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:14.543989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.543965    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-dshm" (OuterVolumeSpecName: "dshm") pod "4ece20b2-c1d2-4243-ad32-ea6881c22699" (UID: "4ece20b2-c1d2-4243-ad32-ea6881c22699"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:37:14.544097 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.544000    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ece20b2-c1d2-4243-ad32-ea6881c22699-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ece20b2-c1d2-4243-ad32-ea6881c22699" (UID: "4ece20b2-c1d2-4243-ad32-ea6881c22699"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:37:14.544097 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.544039    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ece20b2-c1d2-4243-ad32-ea6881c22699-kube-api-access-lxmcd" (OuterVolumeSpecName: "kube-api-access-lxmcd") pod "4ece20b2-c1d2-4243-ad32-ea6881c22699" (UID: "4ece20b2-c1d2-4243-ad32-ea6881c22699"). InnerVolumeSpecName "kube-api-access-lxmcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:37:14.642921 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.642890    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ece20b2-c1d2-4243-ad32-ea6881c22699-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:14.642921 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.642918    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxmcd\" (UniqueName: \"kubernetes.io/projected/4ece20b2-c1d2-4243-ad32-ea6881c22699-kube-api-access-lxmcd\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:14.642921 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:14.642929    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ece20b2-c1d2-4243-ad32-ea6881c22699-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:15.072588 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:15.072555    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:15.199354 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:15.199315    2560 generic.go:356] "Generic (PLEG): container finished" podID="4ece20b2-c1d2-4243-ad32-ea6881c22699" containerID="6ea009223bc1cb45619bae56a617e804356b77dd2e4c01c2d1a6a1b678e9f5ac" exitCode=0
Sep 11 17:37:15.199809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:15.199381    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr" event={"ID":"4ece20b2-c1d2-4243-ad32-ea6881c22699","Type":"ContainerDied","Data":"6ea009223bc1cb45619bae56a617e804356b77dd2e4c01c2d1a6a1b678e9f5ac"}
Sep 11 17:37:15.199809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:15.199407    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr" event={"ID":"4ece20b2-c1d2-4243-ad32-ea6881c22699","Type":"ContainerDied","Data":"21dd1da2feeb05f61afd7ea10d5cd11a98fabbd7fd8bd572e80f865682e28d91"}
Sep 11 17:37:15.199809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:15.199423    2560 scope.go:117] "RemoveContainer" containerID="6ea009223bc1cb45619bae56a617e804356b77dd2e4c01c2d1a6a1b678e9f5ac"
Sep 11 17:37:15.199809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:15.199554    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"
Sep 11 17:37:15.213414 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:15.213397    2560 scope.go:117] "RemoveContainer" containerID="6ea009223bc1cb45619bae56a617e804356b77dd2e4c01c2d1a6a1b678e9f5ac"
Sep 11 17:37:15.213703 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:37:15.213682    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea009223bc1cb45619bae56a617e804356b77dd2e4c01c2d1a6a1b678e9f5ac\": container with ID starting with 6ea009223bc1cb45619bae56a617e804356b77dd2e4c01c2d1a6a1b678e9f5ac not found: ID does not exist" containerID="6ea009223bc1cb45619bae56a617e804356b77dd2e4c01c2d1a6a1b678e9f5ac"
Sep 11 17:37:15.213750 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:15.213713    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea009223bc1cb45619bae56a617e804356b77dd2e4c01c2d1a6a1b678e9f5ac"} err="failed to get container status \"6ea009223bc1cb45619bae56a617e804356b77dd2e4c01c2d1a6a1b678e9f5ac\": rpc error: code = NotFound desc = could not find container \"6ea009223bc1cb45619bae56a617e804356b77dd2e4c01c2d1a6a1b678e9f5ac\": container with ID starting with 6ea009223bc1cb45619bae56a617e804356b77dd2e4c01c2d1a6a1b678e9f5ac not found: ID does not exist"
Sep 11 17:37:15.227131 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:15.227103    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"]
Sep 11 17:37:15.227751 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:15.227680    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-87b0dc/explicit-model-87b0dc-kserve-666bf4dfd9-blmnr"]
Sep 11 17:37:16.051372 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:16.051338    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ece20b2-c1d2-4243-ad32-ea6881c22699" path="/var/lib/kubelet/pods/4ece20b2-c1d2-4243-ad32-ea6881c22699/volumes"
Sep 11 17:37:16.075707 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:16.075674    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:16.176912 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:16.176868    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:16.329582 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:16.329492    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:16.774862 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:16.774830    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:17.021243 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.021208    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"]
Sep 11 17:37:17.021688 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.021667    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ece20b2-c1d2-4243-ad32-ea6881c22699" containerName="main"
Sep 11 17:37:17.021688 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.021685    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ece20b2-c1d2-4243-ad32-ea6881c22699" containerName="main"
Sep 11 17:37:17.021846 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.021704    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac3099a-9ad5-4369-a5f0-8394dd851c3b" containerName="revoke-keys"
Sep 11 17:37:17.021846 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.021710    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac3099a-9ad5-4369-a5f0-8394dd851c3b" containerName="revoke-keys"
Sep 11 17:37:17.021846 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.021777    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ece20b2-c1d2-4243-ad32-ea6881c22699" containerName="main"
Sep 11 17:37:17.021846 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.021788    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac3099a-9ad5-4369-a5f0-8394dd851c3b" containerName="revoke-keys"
Sep 11 17:37:17.026784 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.026732    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.031457 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.031430    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-080169-openshift-default-dockercfg-l682d\""
Sep 11 17:37:17.047797 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.047777    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"]
Sep 11 17:37:17.101811 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.101784    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:17.170347 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.170312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istiod-ca-cert\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.170347 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.170348    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-workload-certs\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.170592 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.170385    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-podinfo\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.170592 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.170453    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-envoy\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.170592 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.170549    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vssns\" (UniqueName: \"kubernetes.io/projected/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-kube-api-access-vssns\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.170592 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.170586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-credential-socket\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.170882 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.170860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-token\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.170986 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.170951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-workload-socket\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.171109 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.171014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-data\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.272354 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.272323    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-token\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.272354 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.272355    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-workload-socket\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.272576 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.272375    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-data\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.272576 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.272564    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istiod-ca-cert\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.272667 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.272599    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-workload-certs\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.272667 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.272637    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-podinfo\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.272769 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.272686    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-envoy\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.272835 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.272770    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vssns\" (UniqueName: \"kubernetes.io/projected/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-kube-api-access-vssns\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.272835 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.272802    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-workload-socket\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.272939 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.272832    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-data\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.272939 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.272882    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-credential-socket\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.272939 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.272920    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-workload-certs\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.273206 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.273179    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-credential-socket\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.273300 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.273267    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istiod-ca-cert\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.274964 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.274942    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-envoy\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.275311 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.275291    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-podinfo\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.280626 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.280576    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vssns\" (UniqueName: \"kubernetes.io/projected/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-kube-api-access-vssns\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.280952 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.280932    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-token\") pod \"e2e-noten-080169-openshift-default-85c7795b6d-49hhj\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") " pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.337792 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.337772    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:17.448647 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.448617    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:17.613667 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.613641    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"]
Sep 11 17:37:17.615049 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:37:17.615013    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2044b2_2443_432f_b1ef_9fcc5b0e2b55.slice/crio-c9e69198048ffc121044f120c946a8e82edbc3ad599acb2d8e0e12d19b093e19 WatchSource:0}: Error finding container c9e69198048ffc121044f120c946a8e82edbc3ad599acb2d8e0e12d19b093e19: Status 404 returned error can't find the container with id c9e69198048ffc121044f120c946a8e82edbc3ad599acb2d8e0e12d19b093e19
Sep 11 17:37:17.617356 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.617323    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:37:17.617439 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.617394    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:37:17.617439 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:17.617427    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:37:18.217736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:18.217701    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj" event={"ID":"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55","Type":"ContainerStarted","Data":"b1dd5d6e5949c30634fe55e219b976937bee07f7b4683c3782aa0e28d8229ce0"}
Sep 11 17:37:18.217964 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:18.217764    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj" event={"ID":"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55","Type":"ContainerStarted","Data":"c9e69198048ffc121044f120c946a8e82edbc3ad599acb2d8e0e12d19b093e19"}
Sep 11 17:37:18.240132 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:18.240077    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj" podStartSLOduration=2.240061597 podStartE2EDuration="2.240061597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:37:18.238840693 +0000 UTC m=+1008.671293474" watchObservedRunningTime="2026-09-11 17:37:18.240061597 +0000 UTC m=+1008.672514371"
Sep 11 17:37:18.338577 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:18.338540    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:18.344236 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:18.344211    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:18.523599 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:18.523510    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:18.622885 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:18.622854    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:18.738569 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:18.738537    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:19.223789 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:19.223752    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:19.224768 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:19.224743    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:29.085857 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:29.085819    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:30.519826 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:30.519769    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:31.160134 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:31.160095    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:31.910404 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:31.910365    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:32.626727 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:32.626695    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:34.911744 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:34.911709    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:35.577053 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:35.577021    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:35.677004 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:35.676974    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:36.043247 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:36.043217    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:41.625002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:41.624968    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:41.772383 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:41.772344    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:41.967507 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:41.967462    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2340fb02-nn656"]
Sep 11 17:37:41.972264 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:41.972244    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2340fb02-nn656"
Sep 11 17:37:41.975175 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:41.975151    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:37:41.975297 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:41.975178    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:37:41.981445 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:41.981417    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2340fb02-nn656"]
Sep 11 17:37:42.132974 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:42.132928    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2340fb02-nn656\" (UID: \"b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2340fb02-nn656"
Sep 11 17:37:42.234135 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:42.234030    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2340fb02-nn656\" (UID: \"b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2340fb02-nn656"
Sep 11 17:37:42.234719 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:42.234695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2340fb02-nn656\" (UID: \"b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2340fb02-nn656"
Sep 11 17:37:42.284051 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:42.284010    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2340fb02-nn656"
Sep 11 17:37:42.555007 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:42.554981    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2340fb02-nn656"]
Sep 11 17:37:42.557376 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:37:42.557350    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cc24a2_63c9_4e6c_8ae6_fdbcde1fb646.slice/crio-2c892bdc0324ed5416dcd3d20839c795072a453b9f73f48194070ac3477d2e54 WatchSource:0}: Error finding container 2c892bdc0324ed5416dcd3d20839c795072a453b9f73f48194070ac3477d2e54: Status 404 returned error can't find the container with id 2c892bdc0324ed5416dcd3d20839c795072a453b9f73f48194070ac3477d2e54
Sep 11 17:37:42.825578 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:42.825552    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:43.353963 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:43.353928    2560 generic.go:356] "Generic (PLEG): container finished" podID="b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646" containerID="368ad45e1aea29f3f6936a9801c96e7025cc5dc3870ff7b04dc11af425519e4d" exitCode=0
Sep 11 17:37:43.354140 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:43.353976    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2340fb02-nn656" event={"ID":"b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646","Type":"ContainerDied","Data":"368ad45e1aea29f3f6936a9801c96e7025cc5dc3870ff7b04dc11af425519e4d"}
Sep 11 17:37:43.354140 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:43.354029    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2340fb02-nn656" event={"ID":"b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646","Type":"ContainerStarted","Data":"2c892bdc0324ed5416dcd3d20839c795072a453b9f73f48194070ac3477d2e54"}
Sep 11 17:37:43.466444 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:43.466369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:44.121734 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:44.121699    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:44.485420 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:44.485399    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2340fb02-nn656"
Sep 11 17:37:44.657127 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:44.657089    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646-maas-api-service-ca\") pod \"b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646\" (UID: \"b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646\") "
Sep 11 17:37:44.657567 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:44.657469    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646" (UID: "b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:37:44.758091 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:44.758039    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:44.829099 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:44.829070    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:45.365462 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:45.365427    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2340fb02-nn656" event={"ID":"b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646","Type":"ContainerDied","Data":"2c892bdc0324ed5416dcd3d20839c795072a453b9f73f48194070ac3477d2e54"}
Sep 11 17:37:45.365462 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:45.365453    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2340fb02-nn656"
Sep 11 17:37:45.365879 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:45.365496    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c892bdc0324ed5416dcd3d20839c795072a453b9f73f48194070ac3477d2e54"
Sep 11 17:37:49.430343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.430307    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"]
Sep 11 17:37:49.430909 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.430890    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646" containerName="revoke-keys"
Sep 11 17:37:49.430982 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.430912    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646" containerName="revoke-keys"
Sep 11 17:37:49.431039 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.431012    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646" containerName="revoke-keys"
Sep 11 17:37:49.434054 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.434033    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.437025 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.437006    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-080169\"/\"openshift-service-ca.crt\""
Sep 11 17:37:49.438274 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.438252    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-080169\"/\"default-dockercfg-sw7rt\""
Sep 11 17:37:49.438358 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.438274    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-080169\"/\"noten-model-080169-kserve-self-signed-certs\""
Sep 11 17:37:49.438358 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.438330    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-080169\"/\"kube-root-ca.crt\""
Sep 11 17:37:49.442412 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.442388    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"]
Sep 11 17:37:49.457598 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.457560    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:49.607238 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.607206    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-model-cache\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.607417 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.607240    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba21b85e-5570-40b9-a688-a7f20e85215b-tls-certs\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.607417 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.607272    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffck4\" (UniqueName: \"kubernetes.io/projected/ba21b85e-5570-40b9-a688-a7f20e85215b-kube-api-access-ffck4\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.607417 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.607387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-home\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.607589 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.607453    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-dshm\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.607589 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.607501    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-tmp-dir\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.708914 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.708783    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-model-cache\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.709403 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.709374    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-model-cache\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.709624 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.709187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba21b85e-5570-40b9-a688-a7f20e85215b-tls-certs\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.709624 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.709600    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffck4\" (UniqueName: \"kubernetes.io/projected/ba21b85e-5570-40b9-a688-a7f20e85215b-kube-api-access-ffck4\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.709764 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.709664    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-home\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.709764 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.709754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-dshm\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.709887 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.709778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-tmp-dir\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.710245 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.710220    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-home\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.710554 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.710530    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-tmp-dir\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.712832 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.712806    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-dshm\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.713043 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.713023    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba21b85e-5570-40b9-a688-a7f20e85215b-tls-certs\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.717554 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.717512    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffck4\" (UniqueName: \"kubernetes.io/projected/ba21b85e-5570-40b9-a688-a7f20e85215b-kube-api-access-ffck4\") pod \"noten-model-080169-kserve-6db5df6895-pdp7h\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") " pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:49.743975 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:49.743948    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:50.014172 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:50.014149    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"]
Sep 11 17:37:50.016134 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:37:50.016109    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba21b85e_5570_40b9_a688_a7f20e85215b.slice/crio-d1bbc4975ef0678919d07dd48514d3118dc64c58fae7ef7b8585481c83d46a8b WatchSource:0}: Error finding container d1bbc4975ef0678919d07dd48514d3118dc64c58fae7ef7b8585481c83d46a8b: Status 404 returned error can't find the container with id d1bbc4975ef0678919d07dd48514d3118dc64c58fae7ef7b8585481c83d46a8b
Sep 11 17:37:50.399512 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:50.399469    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h" event={"ID":"ba21b85e-5570-40b9-a688-a7f20e85215b","Type":"ContainerStarted","Data":"67aa5f54993f789535603800b443210f2b13ce811887121681014ca8632d380f"}
Sep 11 17:37:50.399671 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:50.399521    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h" event={"ID":"ba21b85e-5570-40b9-a688-a7f20e85215b","Type":"ContainerStarted","Data":"d1bbc4975ef0678919d07dd48514d3118dc64c58fae7ef7b8585481c83d46a8b"}
Sep 11 17:37:50.417520 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:50.417457    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h" podStartSLOduration=1.417443381 podStartE2EDuration="1.417443381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:37:50.416518424 +0000 UTC m=+1040.848971195" watchObservedRunningTime="2026-09-11 17:37:50.417443381 +0000 UTC m=+1040.849896157"
Sep 11 17:37:51.670632 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:51.670600    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"]
Sep 11 17:37:51.701932 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:51.701899    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:51.785520 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:51.785488    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"]
Sep 11 17:37:51.785726 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:51.785699    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj" podUID="fe2044b2-2443-432f-b1ef-9fcc5b0e2b55" containerName="istio-proxy" containerID="cri-o://b1dd5d6e5949c30634fe55e219b976937bee07f7b4683c3782aa0e28d8229ce0" gracePeriod=30
Sep 11 17:37:51.809629 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:51.809598    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:52.026466 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:52.026350    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:52.151775 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:52.151729    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:52.493986 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:52.493952    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:52.563534 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:52.563509    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:52.623274 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:52.623250    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:52.694440 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:52.694404    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2340fb02-nn656"]
Sep 11 17:37:52.697564 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:52.697536    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2340fb02-nn656"]
Sep 11 17:37:52.697694 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:52.697670    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:52.753631 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:52.753563    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:53.057643 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:53.057567    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:53.156982 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:53.156950    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:53.969849 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:53.969819    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:54.050692 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:54.050653    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646" path="/var/lib/kubelet/pods/b1cc24a2-63c9-4e6c-8ae6-fdbcde1fb646/volumes"
Sep 11 17:37:54.566805 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:54.566776    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:54.808277 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:54.808242    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"]
Sep 11 17:37:54.831120 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:54.831045    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:55.231136 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:55.231103    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:56.018290 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:56.018258    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"]
Sep 11 17:37:56.039007 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:56.038972    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:56.137841 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:56.137809    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f"]
Sep 11 17:37:56.141374 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:56.141359    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f"
Sep 11 17:37:56.143766 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:56.143729    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:37:56.143910 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:56.143798    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:37:56.147774 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:56.147747    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f"]
Sep 11 17:37:56.173254 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:56.173232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7df06a2-33da-4d05-b35c-f9522eb9e29e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f\" (UID: \"f7df06a2-33da-4d05-b35c-f9522eb9e29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f"
Sep 11 17:37:56.273994 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:56.273917    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7df06a2-33da-4d05-b35c-f9522eb9e29e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f\" (UID: \"f7df06a2-33da-4d05-b35c-f9522eb9e29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f"
Sep 11 17:37:56.274524 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:56.274506    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7df06a2-33da-4d05-b35c-f9522eb9e29e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f\" (UID: \"f7df06a2-33da-4d05-b35c-f9522eb9e29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f"
Sep 11 17:37:56.454512 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:56.454460    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f"
Sep 11 17:37:56.724485 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:56.724436    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f"]
Sep 11 17:37:56.727749 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:37:56.727721    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7df06a2_33da_4d05_b35c_f9522eb9e29e.slice/crio-33cfa43a1c2c6cb7bc4094b34907b83a81984ae66c6d6c6040c8fd0e975dd9a1 WatchSource:0}: Error finding container 33cfa43a1c2c6cb7bc4094b34907b83a81984ae66c6d6c6040c8fd0e975dd9a1: Status 404 returned error can't find the container with id 33cfa43a1c2c6cb7bc4094b34907b83a81984ae66c6d6c6040c8fd0e975dd9a1
Sep 11 17:37:56.900939 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:56.900865    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:57.015036 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.014972    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:57.081699 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.081663    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-credential-socket\") pod \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") "
Sep 11 17:37:57.081699 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.081700    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-token\") pod \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") "
Sep 11 17:37:57.081950 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.081766    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-podinfo\") pod \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") "
Sep 11 17:37:57.081950 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.081786    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vssns\" (UniqueName: \"kubernetes.io/projected/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-kube-api-access-vssns\") pod \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") "
Sep 11 17:37:57.081950 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.081812    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-workload-socket\") pod \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") "
Sep 11 17:37:57.081950 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.081840    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istiod-ca-cert\") pod \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") "
Sep 11 17:37:57.082148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.081951    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-workload-certs\") pod \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") "
Sep 11 17:37:57.082148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.082018    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55" (UID: "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:37:57.082148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.082037    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-envoy\") pod \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") "
Sep 11 17:37:57.082148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.082101    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-data\") pod \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\" (UID: \"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55\") "
Sep 11 17:37:57.082352 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.082211    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55" (UID: "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:37:57.082352 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.082275    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55" (UID: "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:37:57.082352 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.082304    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55" (UID: "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:37:57.082538 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.082468    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-data" (OuterVolumeSpecName: "istio-data") pod "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55" (UID: "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:37:57.082659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.082637    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-credential-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:57.082659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.082659    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-workload-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:57.082851 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.082674    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istiod-ca-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:57.082851 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.082687    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-workload-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:57.082851 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.082699    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-data\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:57.084176 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.084148    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-kube-api-access-vssns" (OuterVolumeSpecName: "kube-api-access-vssns") pod "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55" (UID: "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55"). InnerVolumeSpecName "kube-api-access-vssns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:37:57.084259 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.084169    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55" (UID: "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:37:57.084259 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.084178    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55" (UID: "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:37:57.084259 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.084225    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55" (UID: "fe2044b2-2443-432f-b1ef-9fcc5b0e2b55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:37:57.184119 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.184083    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-token\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:57.184119 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.184115    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-podinfo\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:57.184119 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.184126    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vssns\" (UniqueName: \"kubernetes.io/projected/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-kube-api-access-vssns\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:57.184379 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.184136    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55-istio-envoy\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:57.226709 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.226638    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:57.447761 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.447731    2560 generic.go:356] "Generic (PLEG): container finished" podID="fe2044b2-2443-432f-b1ef-9fcc5b0e2b55" containerID="b1dd5d6e5949c30634fe55e219b976937bee07f7b4683c3782aa0e28d8229ce0" exitCode=0
Sep 11 17:37:57.448205 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.447777    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"
Sep 11 17:37:57.448205 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.447813    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj" event={"ID":"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55","Type":"ContainerDied","Data":"b1dd5d6e5949c30634fe55e219b976937bee07f7b4683c3782aa0e28d8229ce0"}
Sep 11 17:37:57.448205 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.447880    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj" event={"ID":"fe2044b2-2443-432f-b1ef-9fcc5b0e2b55","Type":"ContainerDied","Data":"c9e69198048ffc121044f120c946a8e82edbc3ad599acb2d8e0e12d19b093e19"}
Sep 11 17:37:57.448205 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.447914    2560 scope.go:117] "RemoveContainer" containerID="b1dd5d6e5949c30634fe55e219b976937bee07f7b4683c3782aa0e28d8229ce0"
Sep 11 17:37:57.449603 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.449581    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f" event={"ID":"f7df06a2-33da-4d05-b35c-f9522eb9e29e","Type":"ContainerStarted","Data":"c4c1f613559d7e58fd4526e4b2a309b09191f489a76f1cc6f0b8295cfba96d79"}
Sep 11 17:37:57.449688 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.449625    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f" event={"ID":"f7df06a2-33da-4d05-b35c-f9522eb9e29e","Type":"ContainerStarted","Data":"33cfa43a1c2c6cb7bc4094b34907b83a81984ae66c6d6c6040c8fd0e975dd9a1"}
Sep 11 17:37:57.457441 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.457291    2560 scope.go:117] "RemoveContainer" containerID="b1dd5d6e5949c30634fe55e219b976937bee07f7b4683c3782aa0e28d8229ce0"
Sep 11 17:37:57.457592 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:37:57.457570    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1dd5d6e5949c30634fe55e219b976937bee07f7b4683c3782aa0e28d8229ce0\": container with ID starting with b1dd5d6e5949c30634fe55e219b976937bee07f7b4683c3782aa0e28d8229ce0 not found: ID does not exist" containerID="b1dd5d6e5949c30634fe55e219b976937bee07f7b4683c3782aa0e28d8229ce0"
Sep 11 17:37:57.457645 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.457600    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1dd5d6e5949c30634fe55e219b976937bee07f7b4683c3782aa0e28d8229ce0"} err="failed to get container status \"b1dd5d6e5949c30634fe55e219b976937bee07f7b4683c3782aa0e28d8229ce0\": rpc error: code = NotFound desc = could not find container \"b1dd5d6e5949c30634fe55e219b976937bee07f7b4683c3782aa0e28d8229ce0\": container with ID starting with b1dd5d6e5949c30634fe55e219b976937bee07f7b4683c3782aa0e28d8229ce0 not found: ID does not exist"
Sep 11 17:37:57.469686 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.469645    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f" podStartSLOduration=1.469629959 podStartE2EDuration="1.469629959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:37:57.468537232 +0000 UTC m=+1047.900990007" watchObservedRunningTime="2026-09-11 17:37:57.469629959 +0000 UTC m=+1047.902082734"
Sep 11 17:37:57.483379 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.483353    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"]
Sep 11 17:37:57.487542 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.487506    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-080169-openshift-default-85c7795b6d-49hhj"]
Sep 11 17:37:57.720661 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:57.720575    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:58.051738 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:58.051652    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe2044b2-2443-432f-b1ef-9fcc5b0e2b55" path="/var/lib/kubelet/pods/fe2044b2-2443-432f-b1ef-9fcc5b0e2b55/volumes"
Sep 11 17:37:58.361781 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:58.361699    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:58.457322 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:58.457288    2560 generic.go:356] "Generic (PLEG): container finished" podID="f7df06a2-33da-4d05-b35c-f9522eb9e29e" containerID="c4c1f613559d7e58fd4526e4b2a309b09191f489a76f1cc6f0b8295cfba96d79" exitCode=0
Sep 11 17:37:58.457777 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:58.457381    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f" event={"ID":"f7df06a2-33da-4d05-b35c-f9522eb9e29e","Type":"ContainerDied","Data":"c4c1f613559d7e58fd4526e4b2a309b09191f489a76f1cc6f0b8295cfba96d79"}
Sep 11 17:37:59.045667 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:59.045634    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:59.145166 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:59.145133    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:59.298814 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:59.298738    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:59.596227 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:59.596200    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f"
Sep 11 17:37:59.616964 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:59.616724    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:59.710824 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:59.710794    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7df06a2-33da-4d05-b35c-f9522eb9e29e-maas-api-service-ca\") pod \"f7df06a2-33da-4d05-b35c-f9522eb9e29e\" (UID: \"f7df06a2-33da-4d05-b35c-f9522eb9e29e\") "
Sep 11 17:37:59.711111 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:59.711086    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:37:59.711206 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:59.711183    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7df06a2-33da-4d05-b35c-f9522eb9e29e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7df06a2-33da-4d05-b35c-f9522eb9e29e" (UID: "f7df06a2-33da-4d05-b35c-f9522eb9e29e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:37:59.711262 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:59.711248    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7df06a2-33da-4d05-b35c-f9522eb9e29e-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:37:59.744442 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:59.744415    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:37:59.818034 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:37:59.818005    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:00.469301 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:00.469270    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f"
Sep 11 17:38:00.469468 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:00.469301    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f" event={"ID":"f7df06a2-33da-4d05-b35c-f9522eb9e29e","Type":"ContainerDied","Data":"33cfa43a1c2c6cb7bc4094b34907b83a81984ae66c6d6c6040c8fd0e975dd9a1"}
Sep 11 17:38:00.469468 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:00.469342    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33cfa43a1c2c6cb7bc4094b34907b83a81984ae66c6d6c6040c8fd0e975dd9a1"
Sep 11 17:38:05.680546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:05.680468    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:05.780458 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:05.780406    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:05.881608 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:05.881571    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:06.040570 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:06.040467    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:06.799559 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:06.799526    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:06.877411 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:06.877376    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:06.962549 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:06.962519    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:07.010909 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.010870    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f"]
Sep 11 17:38:07.013790 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.013764    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d92fb3-bm55f"]
Sep 11 17:38:07.061062 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.060987    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:07.363630 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.363547    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:07.434783 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.434751    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h" podUID="ba21b85e-5570-40b9-a688-a7f20e85215b" containerName="main" containerID="cri-o://67aa5f54993f789535603800b443210f2b13ce811887121681014ca8632d380f" gracePeriod=45
Sep 11 17:38:07.467158 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.467132    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:07.681655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.681633    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:38:07.793147 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.793118    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-model-cache\") pod \"ba21b85e-5570-40b9-a688-a7f20e85215b\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") "
Sep 11 17:38:07.793147 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.793154    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-home\") pod \"ba21b85e-5570-40b9-a688-a7f20e85215b\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") "
Sep 11 17:38:07.793358 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.793198    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba21b85e-5570-40b9-a688-a7f20e85215b-tls-certs\") pod \"ba21b85e-5570-40b9-a688-a7f20e85215b\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") "
Sep 11 17:38:07.793358 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.793256    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-dshm\") pod \"ba21b85e-5570-40b9-a688-a7f20e85215b\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") "
Sep 11 17:38:07.793358 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.793296    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffck4\" (UniqueName: \"kubernetes.io/projected/ba21b85e-5570-40b9-a688-a7f20e85215b-kube-api-access-ffck4\") pod \"ba21b85e-5570-40b9-a688-a7f20e85215b\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") "
Sep 11 17:38:07.793358 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.793326    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-tmp-dir\") pod \"ba21b85e-5570-40b9-a688-a7f20e85215b\" (UID: \"ba21b85e-5570-40b9-a688-a7f20e85215b\") "
Sep 11 17:38:07.793557 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.793450    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-home" (OuterVolumeSpecName: "home") pod "ba21b85e-5570-40b9-a688-a7f20e85215b" (UID: "ba21b85e-5570-40b9-a688-a7f20e85215b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:07.793557 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.793444    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba21b85e-5570-40b9-a688-a7f20e85215b" (UID: "ba21b85e-5570-40b9-a688-a7f20e85215b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:07.793747 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.793717    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba21b85e-5570-40b9-a688-a7f20e85215b" (UID: "ba21b85e-5570-40b9-a688-a7f20e85215b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:07.793826 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.793801    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:07.793826 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.793819    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:07.795518 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.795486    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba21b85e-5570-40b9-a688-a7f20e85215b-kube-api-access-ffck4" (OuterVolumeSpecName: "kube-api-access-ffck4") pod "ba21b85e-5570-40b9-a688-a7f20e85215b" (UID: "ba21b85e-5570-40b9-a688-a7f20e85215b"). InnerVolumeSpecName "kube-api-access-ffck4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:38:07.795518 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.795504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-dshm" (OuterVolumeSpecName: "dshm") pod "ba21b85e-5570-40b9-a688-a7f20e85215b" (UID: "ba21b85e-5570-40b9-a688-a7f20e85215b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:07.795712 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.795515    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba21b85e-5570-40b9-a688-a7f20e85215b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba21b85e-5570-40b9-a688-a7f20e85215b" (UID: "ba21b85e-5570-40b9-a688-a7f20e85215b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:38:07.895186 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.895156    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:07.895186 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.895183    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffck4\" (UniqueName: \"kubernetes.io/projected/ba21b85e-5570-40b9-a688-a7f20e85215b-kube-api-access-ffck4\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:07.895186 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.895194    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba21b85e-5570-40b9-a688-a7f20e85215b-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:07.895614 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:07.895203    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba21b85e-5570-40b9-a688-a7f20e85215b-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:08.050826 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:08.050746    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7df06a2-33da-4d05-b35c-f9522eb9e29e" path="/var/lib/kubelet/pods/f7df06a2-33da-4d05-b35c-f9522eb9e29e/volumes"
Sep 11 17:38:08.150228 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:08.150199    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:08.515316 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:08.515287    2560 generic.go:356] "Generic (PLEG): container finished" podID="ba21b85e-5570-40b9-a688-a7f20e85215b" containerID="67aa5f54993f789535603800b443210f2b13ce811887121681014ca8632d380f" exitCode=0
Sep 11 17:38:08.515470 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:08.515329    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"
Sep 11 17:38:08.515470 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:08.515375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h" event={"ID":"ba21b85e-5570-40b9-a688-a7f20e85215b","Type":"ContainerDied","Data":"67aa5f54993f789535603800b443210f2b13ce811887121681014ca8632d380f"}
Sep 11 17:38:08.515470 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:08.515418    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h" event={"ID":"ba21b85e-5570-40b9-a688-a7f20e85215b","Type":"ContainerDied","Data":"d1bbc4975ef0678919d07dd48514d3118dc64c58fae7ef7b8585481c83d46a8b"}
Sep 11 17:38:08.515470 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:08.515436    2560 scope.go:117] "RemoveContainer" containerID="67aa5f54993f789535603800b443210f2b13ce811887121681014ca8632d380f"
Sep 11 17:38:08.524124 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:08.524106    2560 scope.go:117] "RemoveContainer" containerID="67aa5f54993f789535603800b443210f2b13ce811887121681014ca8632d380f"
Sep 11 17:38:08.524358 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:38:08.524336    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67aa5f54993f789535603800b443210f2b13ce811887121681014ca8632d380f\": container with ID starting with 67aa5f54993f789535603800b443210f2b13ce811887121681014ca8632d380f not found: ID does not exist" containerID="67aa5f54993f789535603800b443210f2b13ce811887121681014ca8632d380f"
Sep 11 17:38:08.524414 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:08.524370    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67aa5f54993f789535603800b443210f2b13ce811887121681014ca8632d380f"} err="failed to get container status \"67aa5f54993f789535603800b443210f2b13ce811887121681014ca8632d380f\": rpc error: code = NotFound desc = could not find container \"67aa5f54993f789535603800b443210f2b13ce811887121681014ca8632d380f\": container with ID starting with 67aa5f54993f789535603800b443210f2b13ce811887121681014ca8632d380f not found: ID does not exist"
Sep 11 17:38:08.531354 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:08.531332    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"]
Sep 11 17:38:08.535227 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:08.535210    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-080169/noten-model-080169-kserve-6db5df6895-pdp7h"]
Sep 11 17:38:09.827745 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:09.827664    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t" podUID="0795e847-5aad-4747-a6d2-bcd0bd247a82" containerName="main" containerID="cri-o://704478e6941f0d390effee585f5cd85fe219ab10a73611c8f18321299da8b7c4" gracePeriod=45
Sep 11 17:38:10.052522 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.052488    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba21b85e-5570-40b9-a688-a7f20e85215b" path="/var/lib/kubelet/pods/ba21b85e-5570-40b9-a688-a7f20e85215b/volumes"
Sep 11 17:38:10.082101 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.082036    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:38:10.220977 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.220936    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-dshm\") pod \"0795e847-5aad-4747-a6d2-bcd0bd247a82\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") "
Sep 11 17:38:10.221178 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.220988    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-model-cache\") pod \"0795e847-5aad-4747-a6d2-bcd0bd247a82\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") "
Sep 11 17:38:10.221178 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.221008    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0795e847-5aad-4747-a6d2-bcd0bd247a82-tls-certs\") pod \"0795e847-5aad-4747-a6d2-bcd0bd247a82\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") "
Sep 11 17:38:10.221178 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.221046    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-tmp-dir\") pod \"0795e847-5aad-4747-a6d2-bcd0bd247a82\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") "
Sep 11 17:38:10.221178 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.221077    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd6lw\" (UniqueName: \"kubernetes.io/projected/0795e847-5aad-4747-a6d2-bcd0bd247a82-kube-api-access-dd6lw\") pod \"0795e847-5aad-4747-a6d2-bcd0bd247a82\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") "
Sep 11 17:38:10.221378 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.221269    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-home\") pod \"0795e847-5aad-4747-a6d2-bcd0bd247a82\" (UID: \"0795e847-5aad-4747-a6d2-bcd0bd247a82\") "
Sep 11 17:38:10.221378 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.221253    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-model-cache" (OuterVolumeSpecName: "model-cache") pod "0795e847-5aad-4747-a6d2-bcd0bd247a82" (UID: "0795e847-5aad-4747-a6d2-bcd0bd247a82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:10.221378 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.221311    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0795e847-5aad-4747-a6d2-bcd0bd247a82" (UID: "0795e847-5aad-4747-a6d2-bcd0bd247a82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:10.221544 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.221516    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-home" (OuterVolumeSpecName: "home") pod "0795e847-5aad-4747-a6d2-bcd0bd247a82" (UID: "0795e847-5aad-4747-a6d2-bcd0bd247a82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:10.221691 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.221667    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:10.221691 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.221686    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:10.221834 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.221694    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:10.223276 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.223255    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0795e847-5aad-4747-a6d2-bcd0bd247a82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0795e847-5aad-4747-a6d2-bcd0bd247a82" (UID: "0795e847-5aad-4747-a6d2-bcd0bd247a82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:38:10.223345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.223282    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-dshm" (OuterVolumeSpecName: "dshm") pod "0795e847-5aad-4747-a6d2-bcd0bd247a82" (UID: "0795e847-5aad-4747-a6d2-bcd0bd247a82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:10.223345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.223305    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0795e847-5aad-4747-a6d2-bcd0bd247a82-kube-api-access-dd6lw" (OuterVolumeSpecName: "kube-api-access-dd6lw") pod "0795e847-5aad-4747-a6d2-bcd0bd247a82" (UID: "0795e847-5aad-4747-a6d2-bcd0bd247a82"). InnerVolumeSpecName "kube-api-access-dd6lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:38:10.323196 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.323163    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd6lw\" (UniqueName: \"kubernetes.io/projected/0795e847-5aad-4747-a6d2-bcd0bd247a82-kube-api-access-dd6lw\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:10.323196 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.323192    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0795e847-5aad-4747-a6d2-bcd0bd247a82-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:10.323196 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.323202    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0795e847-5aad-4747-a6d2-bcd0bd247a82-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:10.528192 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.528158    2560 generic.go:356] "Generic (PLEG): container finished" podID="0795e847-5aad-4747-a6d2-bcd0bd247a82" containerID="704478e6941f0d390effee585f5cd85fe219ab10a73611c8f18321299da8b7c4" exitCode=0
Sep 11 17:38:10.528354 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.528200    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"
Sep 11 17:38:10.528354 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.528241    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t" event={"ID":"0795e847-5aad-4747-a6d2-bcd0bd247a82","Type":"ContainerDied","Data":"704478e6941f0d390effee585f5cd85fe219ab10a73611c8f18321299da8b7c4"}
Sep 11 17:38:10.528354 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.528281    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t" event={"ID":"0795e847-5aad-4747-a6d2-bcd0bd247a82","Type":"ContainerDied","Data":"0872ac1828dd5581551300b847f05b7439824fe1c99fbeb33dff8747b41c9174"}
Sep 11 17:38:10.528354 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.528301    2560 scope.go:117] "RemoveContainer" containerID="704478e6941f0d390effee585f5cd85fe219ab10a73611c8f18321299da8b7c4"
Sep 11 17:38:10.541075 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.541057    2560 scope.go:117] "RemoveContainer" containerID="704478e6941f0d390effee585f5cd85fe219ab10a73611c8f18321299da8b7c4"
Sep 11 17:38:10.541351 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:38:10.541334    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704478e6941f0d390effee585f5cd85fe219ab10a73611c8f18321299da8b7c4\": container with ID starting with 704478e6941f0d390effee585f5cd85fe219ab10a73611c8f18321299da8b7c4 not found: ID does not exist" containerID="704478e6941f0d390effee585f5cd85fe219ab10a73611c8f18321299da8b7c4"
Sep 11 17:38:10.541406 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.541360    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704478e6941f0d390effee585f5cd85fe219ab10a73611c8f18321299da8b7c4"} err="failed to get container status \"704478e6941f0d390effee585f5cd85fe219ab10a73611c8f18321299da8b7c4\": rpc error: code = NotFound desc = could not find container \"704478e6941f0d390effee585f5cd85fe219ab10a73611c8f18321299da8b7c4\": container with ID starting with 704478e6941f0d390effee585f5cd85fe219ab10a73611c8f18321299da8b7c4 not found: ID does not exist"
Sep 11 17:38:10.550866 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.550844    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"]
Sep 11 17:38:10.554707 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:10.554687    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d92fb3/test-model-d92fb3-kserve-5f645f8ddf-9pj5t"]
Sep 11 17:38:11.037002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.036959    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472" podUID="8a22fb70-ac37-464c-97bb-96a7b2d44607" containerName="main" containerID="cri-o://cf52bc7f08718c78d27ac882de50540bf8c27e42ea46dac789618f31887f0b8a" gracePeriod=45
Sep 11 17:38:11.284876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.284856    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:38:11.433497 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.433446    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-model-cache\") pod \"8a22fb70-ac37-464c-97bb-96a7b2d44607\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") "
Sep 11 17:38:11.433661 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.433551    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkt24\" (UniqueName: \"kubernetes.io/projected/8a22fb70-ac37-464c-97bb-96a7b2d44607-kube-api-access-lkt24\") pod \"8a22fb70-ac37-464c-97bb-96a7b2d44607\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") "
Sep 11 17:38:11.433661 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.433583    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-dshm\") pod \"8a22fb70-ac37-464c-97bb-96a7b2d44607\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") "
Sep 11 17:38:11.433661 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.433636    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a22fb70-ac37-464c-97bb-96a7b2d44607-tls-certs\") pod \"8a22fb70-ac37-464c-97bb-96a7b2d44607\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") "
Sep 11 17:38:11.433831 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.433684    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-home\") pod \"8a22fb70-ac37-464c-97bb-96a7b2d44607\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") "
Sep 11 17:38:11.433831 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.433703    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a22fb70-ac37-464c-97bb-96a7b2d44607" (UID: "8a22fb70-ac37-464c-97bb-96a7b2d44607"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:11.433831 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.433716    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-tmp-dir\") pod \"8a22fb70-ac37-464c-97bb-96a7b2d44607\" (UID: \"8a22fb70-ac37-464c-97bb-96a7b2d44607\") "
Sep 11 17:38:11.433985 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.433927    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-home" (OuterVolumeSpecName: "home") pod "8a22fb70-ac37-464c-97bb-96a7b2d44607" (UID: "8a22fb70-ac37-464c-97bb-96a7b2d44607"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:11.433985 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.433946    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a22fb70-ac37-464c-97bb-96a7b2d44607" (UID: "8a22fb70-ac37-464c-97bb-96a7b2d44607"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:11.434155 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.434133    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:11.434155 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.434150    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:11.434155 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.434159    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:11.435800 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.435781    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-dshm" (OuterVolumeSpecName: "dshm") pod "8a22fb70-ac37-464c-97bb-96a7b2d44607" (UID: "8a22fb70-ac37-464c-97bb-96a7b2d44607"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:11.435892 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.435858    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a22fb70-ac37-464c-97bb-96a7b2d44607-kube-api-access-lkt24" (OuterVolumeSpecName: "kube-api-access-lkt24") pod "8a22fb70-ac37-464c-97bb-96a7b2d44607" (UID: "8a22fb70-ac37-464c-97bb-96a7b2d44607"). InnerVolumeSpecName "kube-api-access-lkt24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:38:11.435934 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.435901    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a22fb70-ac37-464c-97bb-96a7b2d44607-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a22fb70-ac37-464c-97bb-96a7b2d44607" (UID: "8a22fb70-ac37-464c-97bb-96a7b2d44607"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:38:11.534747 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.534720    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkt24\" (UniqueName: \"kubernetes.io/projected/8a22fb70-ac37-464c-97bb-96a7b2d44607-kube-api-access-lkt24\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:11.534747 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.534741    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a22fb70-ac37-464c-97bb-96a7b2d44607-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:11.534747 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.534751    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a22fb70-ac37-464c-97bb-96a7b2d44607-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:11.535036 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.534759    2560 generic.go:356] "Generic (PLEG): container finished" podID="8a22fb70-ac37-464c-97bb-96a7b2d44607" containerID="cf52bc7f08718c78d27ac882de50540bf8c27e42ea46dac789618f31887f0b8a" exitCode=0
Sep 11 17:38:11.535036 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.534815    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"
Sep 11 17:38:11.535036 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.534832    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472" event={"ID":"8a22fb70-ac37-464c-97bb-96a7b2d44607","Type":"ContainerDied","Data":"cf52bc7f08718c78d27ac882de50540bf8c27e42ea46dac789618f31887f0b8a"}
Sep 11 17:38:11.535036 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.534865    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472" event={"ID":"8a22fb70-ac37-464c-97bb-96a7b2d44607","Type":"ContainerDied","Data":"c3b47b8d20c53d0cd9201c9763c1d6c6afc9502916b2a1a09926327f0cc76100"}
Sep 11 17:38:11.535036 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.534880    2560 scope.go:117] "RemoveContainer" containerID="cf52bc7f08718c78d27ac882de50540bf8c27e42ea46dac789618f31887f0b8a"
Sep 11 17:38:11.543635 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.543619    2560 scope.go:117] "RemoveContainer" containerID="cf52bc7f08718c78d27ac882de50540bf8c27e42ea46dac789618f31887f0b8a"
Sep 11 17:38:11.543920 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:38:11.543904    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf52bc7f08718c78d27ac882de50540bf8c27e42ea46dac789618f31887f0b8a\": container with ID starting with cf52bc7f08718c78d27ac882de50540bf8c27e42ea46dac789618f31887f0b8a not found: ID does not exist" containerID="cf52bc7f08718c78d27ac882de50540bf8c27e42ea46dac789618f31887f0b8a"
Sep 11 17:38:11.543992 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.543928    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf52bc7f08718c78d27ac882de50540bf8c27e42ea46dac789618f31887f0b8a"} err="failed to get container status \"cf52bc7f08718c78d27ac882de50540bf8c27e42ea46dac789618f31887f0b8a\": rpc error: code = NotFound desc = could not find container \"cf52bc7f08718c78d27ac882de50540bf8c27e42ea46dac789618f31887f0b8a\": container with ID starting with cf52bc7f08718c78d27ac882de50540bf8c27e42ea46dac789618f31887f0b8a not found: ID does not exist"
Sep 11 17:38:11.556249 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.556224    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"]
Sep 11 17:38:11.560469 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.560448    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d148f3/test-model-d148f3-kserve-5cb8d65df-x2472"]
Sep 11 17:38:11.858272 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.858177    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:11.958116 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:11.958075    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:12.051112 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:12.051079    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0795e847-5aad-4747-a6d2-bcd0bd247a82" path="/var/lib/kubelet/pods/0795e847-5aad-4747-a6d2-bcd0bd247a82/volumes"
Sep 11 17:38:12.051555 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:12.051540    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a22fb70-ac37-464c-97bb-96a7b2d44607" path="/var/lib/kubelet/pods/8a22fb70-ac37-464c-97bb-96a7b2d44607/volumes"
Sep 11 17:38:12.737288 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:12.737256    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:13.003677 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.003594    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"]
Sep 11 17:38:13.004003 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.003990    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7df06a2-33da-4d05-b35c-f9522eb9e29e" containerName="revoke-keys"
Sep 11 17:38:13.004060 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.004005    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7df06a2-33da-4d05-b35c-f9522eb9e29e" containerName="revoke-keys"
Sep 11 17:38:13.004060 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.004014    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe2044b2-2443-432f-b1ef-9fcc5b0e2b55" containerName="istio-proxy"
Sep 11 17:38:13.004060 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.004019    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe2044b2-2443-432f-b1ef-9fcc5b0e2b55" containerName="istio-proxy"
Sep 11 17:38:13.004060 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.004030    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba21b85e-5570-40b9-a688-a7f20e85215b" containerName="main"
Sep 11 17:38:13.004060 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.004036    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba21b85e-5570-40b9-a688-a7f20e85215b" containerName="main"
Sep 11 17:38:13.004060 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.004043    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a22fb70-ac37-464c-97bb-96a7b2d44607" containerName="main"
Sep 11 17:38:13.004060 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.004048    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a22fb70-ac37-464c-97bb-96a7b2d44607" containerName="main"
Sep 11 17:38:13.004060 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.004055    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0795e847-5aad-4747-a6d2-bcd0bd247a82" containerName="main"
Sep 11 17:38:13.004060 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.004060    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0795e847-5aad-4747-a6d2-bcd0bd247a82" containerName="main"
Sep 11 17:38:13.004423 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.004125    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0795e847-5aad-4747-a6d2-bcd0bd247a82" containerName="main"
Sep 11 17:38:13.004423 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.004131    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7df06a2-33da-4d05-b35c-f9522eb9e29e" containerName="revoke-keys"
Sep 11 17:38:13.004423 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.004140    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba21b85e-5570-40b9-a688-a7f20e85215b" containerName="main"
Sep 11 17:38:13.004423 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.004147    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a22fb70-ac37-464c-97bb-96a7b2d44607" containerName="main"
Sep 11 17:38:13.004423 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.004153    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe2044b2-2443-432f-b1ef-9fcc5b0e2b55" containerName="istio-proxy"
Sep 11 17:38:13.008874 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.008854    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.011889 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.011870    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-470521-openshift-default-dockercfg-84ngz\""
Sep 11 17:38:13.022343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.022319    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"]
Sep 11 17:38:13.042984 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.042945    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:13.139333 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.139303    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:13.151558 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.151526    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-workload-certs\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.151725 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.151565    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-workload-socket\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.151725 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.151589    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616deae5-3a6f-46c5-8c5f-63ab3474767f-istiod-ca-cert\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.151725 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.151698    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqcx5\" (UniqueName: \"kubernetes.io/projected/616deae5-3a6f-46c5-8c5f-63ab3474767f-kube-api-access-pqcx5\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.151898 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.151801    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-credential-socket\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.151898 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.151832    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-podinfo\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.151898 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.151881    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-token\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.152048 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.151922    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-envoy\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.152048 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.151965    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-data\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.253271 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.253236    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-workload-certs\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.253271 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.253276    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-workload-socket\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.253655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.253300    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616deae5-3a6f-46c5-8c5f-63ab3474767f-istiod-ca-cert\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.253655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.253322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqcx5\" (UniqueName: \"kubernetes.io/projected/616deae5-3a6f-46c5-8c5f-63ab3474767f-kube-api-access-pqcx5\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.253655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.253359    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-credential-socket\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.253655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.253375    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-podinfo\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.253655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.253404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-token\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.253655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.253547    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-envoy\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.253655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.253613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-data\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.253976 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.253747    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-workload-certs\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.253976 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.253802    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-workload-socket\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.253976 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.253956    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-credential-socket\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.254090 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.253973    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-data\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.254090 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.254039    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616deae5-3a6f-46c5-8c5f-63ab3474767f-istiod-ca-cert\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.255870 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.255849    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-envoy\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.256104 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.256086    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-podinfo\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.261789 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.261759    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-token\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.261938 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.261918    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqcx5\" (UniqueName: \"kubernetes.io/projected/616deae5-3a6f-46c5-8c5f-63ab3474767f-kube-api-access-pqcx5\") pod \"e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") " pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.320046 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.320012    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:13.542904 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.542819    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:13.602434 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.602395    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"]
Sep 11 17:38:13.603232 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:38:13.603204    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616deae5_3a6f_46c5_8c5f_63ab3474767f.slice/crio-de9239484054fa611f1fd607f53d59c640294f430bcf83ac242fd2ff6ff6a627 WatchSource:0}: Error finding container de9239484054fa611f1fd607f53d59c640294f430bcf83ac242fd2ff6ff6a627: Status 404 returned error can't find the container with id de9239484054fa611f1fd607f53d59c640294f430bcf83ac242fd2ff6ff6a627
Sep 11 17:38:13.605722 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.605680    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:38:13.605812 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.605763    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:38:13.605812 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.605805    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:38:13.644351 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.644312    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:13.778708 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:13.778679    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:14.093756 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:14.093727    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b"]
Sep 11 17:38:14.097100 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:14.097080    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b"
Sep 11 17:38:14.099701 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:14.099673    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:38:14.099701 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:14.099690    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:38:14.105046 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:14.105027    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b"]
Sep 11 17:38:14.262647 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:14.262610    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f712860-35be-4fd2-b5b7-b5855b4c7521-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b\" (UID: \"5f712860-35be-4fd2-b5b7-b5855b4c7521\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b"
Sep 11 17:38:14.363342 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:14.363260    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f712860-35be-4fd2-b5b7-b5855b4c7521-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b\" (UID: \"5f712860-35be-4fd2-b5b7-b5855b4c7521\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b"
Sep 11 17:38:14.363962 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:14.363942    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f712860-35be-4fd2-b5b7-b5855b4c7521-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b\" (UID: \"5f712860-35be-4fd2-b5b7-b5855b4c7521\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b"
Sep 11 17:38:14.409409 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:14.409376    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b"
Sep 11 17:38:14.554972 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:14.554938    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b" event={"ID":"616deae5-3a6f-46c5-8c5f-63ab3474767f","Type":"ContainerStarted","Data":"b5def5d2cc566fef14185e06536a0239091db09c7e32bfe0f597bea7654cb0c3"}
Sep 11 17:38:14.555133 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:14.554987    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b" event={"ID":"616deae5-3a6f-46c5-8c5f-63ab3474767f","Type":"ContainerStarted","Data":"de9239484054fa611f1fd607f53d59c640294f430bcf83ac242fd2ff6ff6a627"}
Sep 11 17:38:14.576403 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:14.576354    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b" podStartSLOduration=2.576338382 podStartE2EDuration="2.576338382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:38:14.574635155 +0000 UTC m=+1065.007087957" watchObservedRunningTime="2026-09-11 17:38:14.576338382 +0000 UTC m=+1065.008791159"
Sep 11 17:38:14.678062 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:14.678038    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b"]
Sep 11 17:38:14.679584 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:38:14.679555    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f712860_35be_4fd2_b5b7_b5855b4c7521.slice/crio-a592c49f814c0ed2f58fae123fe4de09166ffa5264e88da65edad859048124ac WatchSource:0}: Error finding container a592c49f814c0ed2f58fae123fe4de09166ffa5264e88da65edad859048124ac: Status 404 returned error can't find the container with id a592c49f814c0ed2f58fae123fe4de09166ffa5264e88da65edad859048124ac
Sep 11 17:38:15.037669 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:15.037637    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:15.320400 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:15.320319    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:15.324902 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:15.324881    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:15.561499 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:15.561441    2560 generic.go:356] "Generic (PLEG): container finished" podID="5f712860-35be-4fd2-b5b7-b5855b4c7521" containerID="68d80e65009815b7f38410428f22d1cda798063fe0f0833539d73335e078b31f" exitCode=0
Sep 11 17:38:15.561694 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:15.561533    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b" event={"ID":"5f712860-35be-4fd2-b5b7-b5855b4c7521","Type":"ContainerDied","Data":"68d80e65009815b7f38410428f22d1cda798063fe0f0833539d73335e078b31f"}
Sep 11 17:38:15.561694 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:15.561588    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b" event={"ID":"5f712860-35be-4fd2-b5b7-b5855b4c7521","Type":"ContainerStarted","Data":"a592c49f814c0ed2f58fae123fe4de09166ffa5264e88da65edad859048124ac"}
Sep 11 17:38:15.561840 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:15.561826    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:15.562692 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:15.562670    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:15.933543 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:15.933500    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:16.573757 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:16.573724    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:16.671001 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:16.670946    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:16.707634 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:16.707610    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b"
Sep 11 17:38:16.798757 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:16.798707    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:16.886734 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:16.886709    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f712860-35be-4fd2-b5b7-b5855b4c7521-maas-api-service-ca\") pod \"5f712860-35be-4fd2-b5b7-b5855b4c7521\" (UID: \"5f712860-35be-4fd2-b5b7-b5855b4c7521\") "
Sep 11 17:38:16.887078 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:16.887058    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f712860-35be-4fd2-b5b7-b5855b4c7521-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f712860-35be-4fd2-b5b7-b5855b4c7521" (UID: "5f712860-35be-4fd2-b5b7-b5855b4c7521"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:38:16.988166 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:16.988134    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f712860-35be-4fd2-b5b7-b5855b4c7521-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:17.576119 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:17.576090    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b"
Sep 11 17:38:17.576119 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:17.576100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b" event={"ID":"5f712860-35be-4fd2-b5b7-b5855b4c7521","Type":"ContainerDied","Data":"a592c49f814c0ed2f58fae123fe4de09166ffa5264e88da65edad859048124ac"}
Sep 11 17:38:17.576564 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:17.576151    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a592c49f814c0ed2f58fae123fe4de09166ffa5264e88da65edad859048124ac"
Sep 11 17:38:17.849893 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:17.849805    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:17.926731 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:17.926698    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:18.289854 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:18.289818    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:18.429210 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:18.429174    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:18.507435 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:18.507405    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:24.221597 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:24.221501    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:24.773059 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:24.773030    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:24.845390 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:24.845357    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:24.932208 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:24.932176    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:24.961397 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:24.961349    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b"]
Sep 11 17:38:24.969769 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:24.969738    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d148f3-zc69b"]
Sep 11 17:38:24.993635 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:24.993601    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:25.326049 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:25.326006    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:25.424291 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:25.424257    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:26.052392 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:26.052360    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f712860-35be-4fd2-b5b7-b5855b4c7521" path="/var/lib/kubelet/pods/5f712860-35be-4fd2-b5b7-b5855b4c7521/volumes"
Sep 11 17:38:26.438659 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:26.438625    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:27.369361 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:27.369332    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:27.934907 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:27.934876    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:28.775405 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:28.775372    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:30.881954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:30.881921    2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:34988->10.133.0.62:15021: read: connection reset by peer" start-of-body=
Sep 11 17:38:30.882541 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:30.882006    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b" podUID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:34988->10.133.0.62:15021: read: connection reset by peer"
Sep 11 17:38:31.144218 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:31.144148    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:31.274387 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:31.274358    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:31.385618 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:31.385585    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:31.511586 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:31.511496    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:31.534311 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:31.534282    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"]
Sep 11 17:38:31.534535 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:31.534511    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q" podUID="a7ee7f20-a157-4d8e-99db-0a37d55febe7" containerName="istio-proxy" containerID="cri-o://27d3da6ad1d165a4f08714e9c5d88b919384603b6eae776cae5d52abe8881484" gracePeriod=30
Sep 11 17:38:31.589162 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:31.589128    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:31.668777 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:31.668747    2560 generic.go:356] "Generic (PLEG): container finished" podID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerID="b5def5d2cc566fef14185e06536a0239091db09c7e32bfe0f597bea7654cb0c3" exitCode=0
Sep 11 17:38:31.668961 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:31.668823    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b" event={"ID":"616deae5-3a6f-46c5-8c5f-63ab3474767f","Type":"ContainerDied","Data":"b5def5d2cc566fef14185e06536a0239091db09c7e32bfe0f597bea7654cb0c3"}
Sep 11 17:38:31.669204 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:31.669190    2560 scope.go:117] "RemoveContainer" containerID="b5def5d2cc566fef14185e06536a0239091db09c7e32bfe0f597bea7654cb0c3"
Sep 11 17:38:31.670173 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:31.670148    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:38:31.670235 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:31.670210    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:38:31.670275 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:31.670240    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:38:31.810000 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:31.809863    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:32.073132 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:32.070939    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:32.140207 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:32.140162    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:32.388028 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:32.387995    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:32.676347 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:32.676266    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b" event={"ID":"616deae5-3a6f-46c5-8c5f-63ab3474767f","Type":"ContainerStarted","Data":"c6ca24933c2e9a5a018081d2b5e9e7fc2472e73d1f5b273e587f1ad8c5316349"}
Sep 11 17:38:33.320488 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:33.320435    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:33.321521 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:33.321492    2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 11 17:38:33.321737 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:33.321713    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b" podUID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 11 17:38:33.447222 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:33.447167    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:33.741750 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:33.741702    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:34.321187 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:34.321142    2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 11 17:38:34.321732 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:34.321216    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b" podUID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 11 17:38:34.389642 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:34.389604    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:34.484452 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:34.484418    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:34.585922 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:34.585820    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:34.953131 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:34.953095    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:35.321253 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:35.321153    2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 11 17:38:35.321253 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:35.321220    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b" podUID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 11 17:38:35.539567 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:35.539540    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:35.640867 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:35.640838    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:35.743942 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:35.743905    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:36.321232 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.321193    2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 11 17:38:36.321596 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.321560    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b" podUID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 11 17:38:36.555158 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.555130    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:36.699604 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.699572    2560 generic.go:356] "Generic (PLEG): container finished" podID="a7ee7f20-a157-4d8e-99db-0a37d55febe7" containerID="27d3da6ad1d165a4f08714e9c5d88b919384603b6eae776cae5d52abe8881484" exitCode=0
Sep 11 17:38:36.699768 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.699642    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q" event={"ID":"a7ee7f20-a157-4d8e-99db-0a37d55febe7","Type":"ContainerDied","Data":"27d3da6ad1d165a4f08714e9c5d88b919384603b6eae776cae5d52abe8881484"}
Sep 11 17:38:36.699768 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.699715    2560 scope.go:117] "RemoveContainer" containerID="9e846049f878608ad44cedf07ef2e2d426bfcc221bd11f238721b2d221ce34ce"
Sep 11 17:38:36.834854 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.834826    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:38:36.894765 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.894734    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-podinfo\") pod \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") "
Sep 11 17:38:36.894765 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.894772    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-data\") pod \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") "
Sep 11 17:38:36.894989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.894801    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-workload-socket\") pod \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") "
Sep 11 17:38:36.894989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.894835    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-workload-certs\") pod \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") "
Sep 11 17:38:36.894989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.894870    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istiod-ca-cert\") pod \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") "
Sep 11 17:38:36.894989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.894894    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-credential-socket\") pod \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") "
Sep 11 17:38:36.895566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.894998    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-token\") pod \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") "
Sep 11 17:38:36.895566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.895054    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2m6d\" (UniqueName: \"kubernetes.io/projected/a7ee7f20-a157-4d8e-99db-0a37d55febe7-kube-api-access-l2m6d\") pod \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") "
Sep 11 17:38:36.895566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.895070    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7ee7f20-a157-4d8e-99db-0a37d55febe7" (UID: "a7ee7f20-a157-4d8e-99db-0a37d55febe7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:36.895566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.895088    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-envoy\") pod \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\" (UID: \"a7ee7f20-a157-4d8e-99db-0a37d55febe7\") "
Sep 11 17:38:36.895566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.895099    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7ee7f20-a157-4d8e-99db-0a37d55febe7" (UID: "a7ee7f20-a157-4d8e-99db-0a37d55febe7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:36.895566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.895219    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7ee7f20-a157-4d8e-99db-0a37d55febe7" (UID: "a7ee7f20-a157-4d8e-99db-0a37d55febe7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:36.895566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.895291    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7ee7f20-a157-4d8e-99db-0a37d55febe7" (UID: "a7ee7f20-a157-4d8e-99db-0a37d55febe7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:38:36.895566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.895511    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7ee7f20-a157-4d8e-99db-0a37d55febe7" (UID: "a7ee7f20-a157-4d8e-99db-0a37d55febe7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:36.896012 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.895577    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-data\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:36.896012 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.895593    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-workload-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:36.896012 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.895602    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-workload-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:36.896012 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.895617    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istiod-ca-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:36.897218 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.897189    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7ee7f20-a157-4d8e-99db-0a37d55febe7" (UID: "a7ee7f20-a157-4d8e-99db-0a37d55febe7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:38:36.897323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.897237    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7ee7f20-a157-4d8e-99db-0a37d55febe7" (UID: "a7ee7f20-a157-4d8e-99db-0a37d55febe7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:38:36.897323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.897266    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7ee7f20-a157-4d8e-99db-0a37d55febe7" (UID: "a7ee7f20-a157-4d8e-99db-0a37d55febe7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:38:36.897408 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.897337    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ee7f20-a157-4d8e-99db-0a37d55febe7-kube-api-access-l2m6d" (OuterVolumeSpecName: "kube-api-access-l2m6d") pod "a7ee7f20-a157-4d8e-99db-0a37d55febe7" (UID: "a7ee7f20-a157-4d8e-99db-0a37d55febe7"). InnerVolumeSpecName "kube-api-access-l2m6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:38:36.996173 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.996134    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-token\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:36.996173 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.996168    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2m6d\" (UniqueName: \"kubernetes.io/projected/a7ee7f20-a157-4d8e-99db-0a37d55febe7-kube-api-access-l2m6d\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:36.996173 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.996179    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-envoy\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:36.996428 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.996190    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ee7f20-a157-4d8e-99db-0a37d55febe7-istio-podinfo\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:36.996428 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:36.996198    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee7f20-a157-4d8e-99db-0a37d55febe7-credential-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:38:37.321018 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:37.320913    2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 11 17:38:37.321018 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:37.320992    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b" podUID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 11 17:38:37.527502 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:37.527015    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:37.622076 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:37.621989    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:37.710910 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:37.710881    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"
Sep 11 17:38:37.710910 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:37.710889    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q" event={"ID":"a7ee7f20-a157-4d8e-99db-0a37d55febe7","Type":"ContainerDied","Data":"3709e11336f6c0b43340fbed96ec6c7457a9f4ee8641b852cd575a6d9e10f87a"}
Sep 11 17:38:37.711164 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:37.710941    2560 scope.go:117] "RemoveContainer" containerID="27d3da6ad1d165a4f08714e9c5d88b919384603b6eae776cae5d52abe8881484"
Sep 11 17:38:37.722448 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:37.722418    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:37.739323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:37.739286    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"]
Sep 11 17:38:37.757502 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:37.757445    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d148f3-openshift-default-7b8747f544-s7d9q"]
Sep 11 17:38:38.051455 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:38.051420    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7ee7f20-a157-4d8e-99db-0a37d55febe7" path="/var/lib/kubelet/pods/a7ee7f20-a157-4d8e-99db-0a37d55febe7/volumes"
Sep 11 17:38:38.068891 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:38.068861    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:38.169292 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:38.169231    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:38.324684 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:38.324604    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:38.324929 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:38.324895    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:38.325686 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:38.325669    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:38:38.537990 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:38.537951    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:38.619496 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:38.619392    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:38.836023 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:38.835977    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:39.065681 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:39.065648    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:39.165049 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:39.165017    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:39.490363 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:39.490325    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:40.361817 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:40.361785    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:41.019947 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:41.019918    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:58.648439 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:58.648335    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:59.106822 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:59.106737    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:38:59.927565 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:38:59.927529    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:00.714278 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:00.714243    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:01.456012 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:01.455970    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:02.237689 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:02.237657    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:02.878643 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:02.878610    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:03.009443 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:03.009410    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:03.664046 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:03.664016    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:04.484914 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:04.484878    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:05.213380 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.213346    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7726-08d62b"]
Sep 11 17:39:05.213718 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.213705    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ee7f20-a157-4d8e-99db-0a37d55febe7" containerName="istio-proxy"
Sep 11 17:39:05.213772 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.213720    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ee7f20-a157-4d8e-99db-0a37d55febe7" containerName="istio-proxy"
Sep 11 17:39:05.213772 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.213738    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f712860-35be-4fd2-b5b7-b5855b4c7521" containerName="revoke-keys"
Sep 11 17:39:05.213772 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.213744    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f712860-35be-4fd2-b5b7-b5855b4c7521" containerName="revoke-keys"
Sep 11 17:39:05.213772 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.213758    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ee7f20-a157-4d8e-99db-0a37d55febe7" containerName="istio-proxy"
Sep 11 17:39:05.213772 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.213764    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ee7f20-a157-4d8e-99db-0a37d55febe7" containerName="istio-proxy"
Sep 11 17:39:05.213921 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.213821    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ee7f20-a157-4d8e-99db-0a37d55febe7" containerName="istio-proxy"
Sep 11 17:39:05.213921 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.213829    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f712860-35be-4fd2-b5b7-b5855b4c7521" containerName="revoke-keys"
Sep 11 17:39:05.213921 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.213836    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ee7f20-a157-4d8e-99db-0a37d55febe7" containerName="istio-proxy"
Sep 11 17:39:05.217663 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.217646    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-08d62b"
Sep 11 17:39:05.220883 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.220858    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jtt66\""
Sep 11 17:39:05.222378 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.222353    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7726-08d62b"]
Sep 11 17:39:05.270195 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.270161    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnjhx\" (UniqueName: \"kubernetes.io/projected/e97df258-1e45-42b2-ae4c-1c91d93ca6cb-kube-api-access-vnjhx\") pod \"test-curl-7726-08d62b\" (UID: \"e97df258-1e45-42b2-ae4c-1c91d93ca6cb\") " pod="opendatahub/test-curl-7726-08d62b"
Sep 11 17:39:05.370838 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.370801    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnjhx\" (UniqueName: \"kubernetes.io/projected/e97df258-1e45-42b2-ae4c-1c91d93ca6cb-kube-api-access-vnjhx\") pod \"test-curl-7726-08d62b\" (UID: \"e97df258-1e45-42b2-ae4c-1c91d93ca6cb\") " pod="opendatahub/test-curl-7726-08d62b"
Sep 11 17:39:05.380771 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.380736    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnjhx\" (UniqueName: \"kubernetes.io/projected/e97df258-1e45-42b2-ae4c-1c91d93ca6cb-kube-api-access-vnjhx\") pod \"test-curl-7726-08d62b\" (UID: \"e97df258-1e45-42b2-ae4c-1c91d93ca6cb\") " pod="opendatahub/test-curl-7726-08d62b"
Sep 11 17:39:05.527490 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.527382    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-08d62b"
Sep 11 17:39:05.535847 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.535814    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:05.795560 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.795535    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7726-08d62b"]
Sep 11 17:39:05.798005 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:39:05.797979    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode97df258_1e45_42b2_ae4c_1c91d93ca6cb.slice/crio-3d68e836ba5748dcb7f43139a22e099aee6bc858a0bcac85638ddbbb115f4354 WatchSource:0}: Error finding container 3d68e836ba5748dcb7f43139a22e099aee6bc858a0bcac85638ddbbb115f4354: Status 404 returned error can't find the container with id 3d68e836ba5748dcb7f43139a22e099aee6bc858a0bcac85638ddbbb115f4354
Sep 11 17:39:05.875304 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.875272    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7726-08d62b" event={"ID":"e97df258-1e45-42b2-ae4c-1c91d93ca6cb","Type":"ContainerStarted","Data":"f8f94ba664fa39442bdab0bf3ec890338d66acd0c9ee873622e6ecb2f99d240b"}
Sep 11 17:39:05.875304 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.875314    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7726-08d62b" event={"ID":"e97df258-1e45-42b2-ae4c-1c91d93ca6cb","Type":"ContainerStarted","Data":"3d68e836ba5748dcb7f43139a22e099aee6bc858a0bcac85638ddbbb115f4354"}
Sep 11 17:39:05.877366 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.877344    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:05.891149 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:05.891096    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7726-08d62b" podStartSLOduration=0.891081629 podStartE2EDuration="891.081629ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:05.890276533 +0000 UTC m=+1116.322729310" watchObservedRunningTime="2026-09-11 17:39:05.891081629 +0000 UTC m=+1116.323534405"
Sep 11 17:39:06.133298 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.133264    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"]
Sep 11 17:39:06.136194 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.136168    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.138914 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.138890    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-db95fd-openshift-default-dockercfg-6zhq7\""
Sep 11 17:39:06.149437 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.149408    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"]
Sep 11 17:39:06.160055 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.160021    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:06.176170 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.176127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-credential-socket\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.176366 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.176181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-podinfo\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.176366 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.176205    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-envoy\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.176366 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.176232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-workload-socket\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.176366 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.176302    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-token\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.176366 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.176344    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-data\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.176718 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.176398    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istiod-ca-cert\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.176718 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.176458    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-workload-certs\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.176718 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.176499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgh66\" (UniqueName: \"kubernetes.io/projected/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-kube-api-access-kgh66\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.260577 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.260533    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:06.277063 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.277030    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-workload-socket\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.277226 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.277075    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-token\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.277226 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.277095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-data\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.277226 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.277181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istiod-ca-cert\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.277226 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.277219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-workload-certs\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.277450 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.277254    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgh66\" (UniqueName: \"kubernetes.io/projected/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-kube-api-access-kgh66\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.277450 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.277287    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-credential-socket\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.277450 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.277337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-podinfo\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.277450 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.277366    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-envoy\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.277450 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.277441    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-workload-socket\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.277704 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.277556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-data\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.277704 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.277624    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-workload-certs\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.277771 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.277711    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-credential-socket\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.277964 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.277944    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istiod-ca-cert\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.279665 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.279645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-envoy\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.279883 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.279867    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-podinfo\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.285730 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.285710    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-token\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.285864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.285847    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgh66\" (UniqueName: \"kubernetes.io/projected/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-kube-api-access-kgh66\") pod \"e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") " pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.446721 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.446620    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:06.489986 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.489930    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7726-08d62b"]
Sep 11 17:39:06.492634 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.492599    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7726-08d62b"]
Sep 11 17:39:06.626078 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.626041    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7726-50c9de"]
Sep 11 17:39:06.626505 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.626463    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e97df258-1e45-42b2-ae4c-1c91d93ca6cb" containerName="test-curl-7726-08d62b"
Sep 11 17:39:06.626505 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.626504    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e97df258-1e45-42b2-ae4c-1c91d93ca6cb" containerName="test-curl-7726-08d62b"
Sep 11 17:39:06.626601 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.626591    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e97df258-1e45-42b2-ae4c-1c91d93ca6cb" containerName="test-curl-7726-08d62b"
Sep 11 17:39:06.629697 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.629680    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-50c9de"
Sep 11 17:39:06.634019 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.633991    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7726-50c9de"]
Sep 11 17:39:06.682148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.682116    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j84p\" (UniqueName: \"kubernetes.io/projected/665156e5-83ea-4bf4-a6b7-bba23b60ab71-kube-api-access-7j84p\") pod \"test-curl-7726-50c9de\" (UID: \"665156e5-83ea-4bf4-a6b7-bba23b60ab71\") " pod="opendatahub/test-curl-7726-50c9de"
Sep 11 17:39:06.728193 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.728166    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"]
Sep 11 17:39:06.730549 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:39:06.730524    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3cfdc6_dc6b_4fe2_8896_42b869d9e202.slice/crio-05dfae5da28bd3404420a1f6078c02f261bec45f5424295c1c41ec7ee01d6b78 WatchSource:0}: Error finding container 05dfae5da28bd3404420a1f6078c02f261bec45f5424295c1c41ec7ee01d6b78: Status 404 returned error can't find the container with id 05dfae5da28bd3404420a1f6078c02f261bec45f5424295c1c41ec7ee01d6b78
Sep 11 17:39:06.733335 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.733302    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:39:06.733396 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.733366    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:39:06.733438 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.733394    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:39:06.783362 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.783330    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j84p\" (UniqueName: \"kubernetes.io/projected/665156e5-83ea-4bf4-a6b7-bba23b60ab71-kube-api-access-7j84p\") pod \"test-curl-7726-50c9de\" (UID: \"665156e5-83ea-4bf4-a6b7-bba23b60ab71\") " pod="opendatahub/test-curl-7726-50c9de"
Sep 11 17:39:06.791703 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.791675    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j84p\" (UniqueName: \"kubernetes.io/projected/665156e5-83ea-4bf4-a6b7-bba23b60ab71-kube-api-access-7j84p\") pod \"test-curl-7726-50c9de\" (UID: \"665156e5-83ea-4bf4-a6b7-bba23b60ab71\") " pod="opendatahub/test-curl-7726-50c9de"
Sep 11 17:39:06.884357 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.884312    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx" event={"ID":"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202","Type":"ContainerStarted","Data":"510e99a01ed52d6c29cae813d2ff0ce2d5ee416a0a4364e014eaf108a7954e2a"}
Sep 11 17:39:06.884533 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.884380    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx" event={"ID":"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202","Type":"ContainerStarted","Data":"05dfae5da28bd3404420a1f6078c02f261bec45f5424295c1c41ec7ee01d6b78"}
Sep 11 17:39:06.884533 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:06.884526    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7726-08d62b" container="test-curl-7726-08d62b"
Sep 11 17:39:06.906764 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.906713    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx" podStartSLOduration=0.906697829 podStartE2EDuration="906.697829ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:06.905553053 +0000 UTC m=+1117.338005831" watchObservedRunningTime="2026-09-11 17:39:06.906697829 +0000 UTC m=+1117.339150606"
Sep 11 17:39:06.940708 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:06.940664    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-50c9de"
Sep 11 17:39:07.215063 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:07.215037    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7726-50c9de"]
Sep 11 17:39:07.217213 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:39:07.217188    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod665156e5_83ea_4bf4_a6b7_bba23b60ab71.slice/crio-ee373577b246cebcd29d4b7c1fe44e5886c6e2753cbffbb3051316160b17499d WatchSource:0}: Error finding container ee373577b246cebcd29d4b7c1fe44e5886c6e2753cbffbb3051316160b17499d: Status 404 returned error can't find the container with id ee373577b246cebcd29d4b7c1fe44e5886c6e2753cbffbb3051316160b17499d
Sep 11 17:39:07.446944 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:07.446903    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:07.452051 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:07.452026    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:07.891715 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:07.891674    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7726-50c9de" event={"ID":"665156e5-83ea-4bf4-a6b7-bba23b60ab71","Type":"ContainerStarted","Data":"68cf270dfbcff73eb6ee35870b0fde446c051fe42c93744b9d57afc05da730a2"}
Sep 11 17:39:07.892148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:07.891724    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7726-50c9de" event={"ID":"665156e5-83ea-4bf4-a6b7-bba23b60ab71","Type":"ContainerStarted","Data":"ee373577b246cebcd29d4b7c1fe44e5886c6e2753cbffbb3051316160b17499d"}
Sep 11 17:39:07.892148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:07.891963    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:07.892148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:07.891970    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7726-08d62b" podUID="e97df258-1e45-42b2-ae4c-1c91d93ca6cb" containerName="test-curl-7726-08d62b" containerID="cri-o://f8f94ba664fa39442bdab0bf3ec890338d66acd0c9ee873622e6ecb2f99d240b" gracePeriod=2
Sep 11 17:39:07.893018 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:07.893003    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:07.909156 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:07.909113    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7726-50c9de" podStartSLOduration=1.909098239 podStartE2EDuration="1.909098239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:07.908527387 +0000 UTC m=+1118.340980165" watchObservedRunningTime="2026-09-11 17:39:07.909098239 +0000 UTC m=+1118.341551089"
Sep 11 17:39:07.932194 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:07.932149    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-08d62b\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="e97df258-1e45-42b2-ae4c-1c91d93ca6cb" pod="opendatahub/test-curl-7726-08d62b"
Sep 11 17:39:08.514628 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:08.514589    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7726-50c9de"]
Sep 11 17:39:08.520986 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:08.520957    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7726-50c9de"]
Sep 11 17:39:08.638618 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:08.638577    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7726-4b363e"]
Sep 11 17:39:08.638956 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:08.638944    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="665156e5-83ea-4bf4-a6b7-bba23b60ab71" containerName="test-curl-7726-50c9de"
Sep 11 17:39:08.639001 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:08.638957    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="665156e5-83ea-4bf4-a6b7-bba23b60ab71" containerName="test-curl-7726-50c9de"
Sep 11 17:39:08.639044 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:08.639034    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="665156e5-83ea-4bf4-a6b7-bba23b60ab71" containerName="test-curl-7726-50c9de"
Sep 11 17:39:08.640937 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:08.640921    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-4b363e"
Sep 11 17:39:08.652115 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:08.652086    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7726-4b363e"]
Sep 11 17:39:08.702185 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:08.702143    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmz8b\" (UniqueName: \"kubernetes.io/projected/4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30-kube-api-access-pmz8b\") pod \"test-curl-7726-4b363e\" (UID: \"4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30\") " pod="opendatahub/test-curl-7726-4b363e"
Sep 11 17:39:08.738359 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:08.738296    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:08.803244 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:08.803200    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmz8b\" (UniqueName: \"kubernetes.io/projected/4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30-kube-api-access-pmz8b\") pod \"test-curl-7726-4b363e\" (UID: \"4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30\") " pod="opendatahub/test-curl-7726-4b363e"
Sep 11 17:39:08.831121 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:08.831085    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmz8b\" (UniqueName: \"kubernetes.io/projected/4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30-kube-api-access-pmz8b\") pod \"test-curl-7726-4b363e\" (UID: \"4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30\") " pod="opendatahub/test-curl-7726-4b363e"
Sep 11 17:39:08.878576 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:08.878533    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:08.897411 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:08.897325    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7726-50c9de" container="test-curl-7726-50c9de"
Sep 11 17:39:08.950860 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:08.950823    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-4b363e"
Sep 11 17:39:09.076157 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:09.076125    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:09.231426 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:09.231344    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7726-4b363e"]
Sep 11 17:39:09.233779 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:39:09.233752    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4716bc1a_8eb8_489b_8ca9_fdb5c68dbc30.slice/crio-0f1d6b868243ca00f2857099b2ca7cde4f613748fa099db59d3b330707e60e0a WatchSource:0}: Error finding container 0f1d6b868243ca00f2857099b2ca7cde4f613748fa099db59d3b330707e60e0a: Status 404 returned error can't find the container with id 0f1d6b868243ca00f2857099b2ca7cde4f613748fa099db59d3b330707e60e0a
Sep 11 17:39:09.905074 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:09.905043    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7726-4b363e" event={"ID":"4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30","Type":"ContainerStarted","Data":"fb6733bdd1739387a87a2df397ae6468baf70ff84421e41bda4e39cac57a0af6"}
Sep 11 17:39:09.905410 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:09.905089    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7726-4b363e" event={"ID":"4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30","Type":"ContainerStarted","Data":"0f1d6b868243ca00f2857099b2ca7cde4f613748fa099db59d3b330707e60e0a"}
Sep 11 17:39:09.905410 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:09.905293    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7726-50c9de" podUID="665156e5-83ea-4bf4-a6b7-bba23b60ab71" containerName="test-curl-7726-50c9de" containerID="cri-o://68cf270dfbcff73eb6ee35870b0fde446c051fe42c93744b9d57afc05da730a2" gracePeriod=2
Sep 11 17:39:09.931001 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:09.930943    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7726-4b363e" podStartSLOduration=1.9309242210000002 podStartE2EDuration="1.930924221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:09.929364521 +0000 UTC m=+1120.361817298" watchObservedRunningTime="2026-09-11 17:39:09.930924221 +0000 UTC m=+1120.363376997"
Sep 11 17:39:09.931884 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:09.931845    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-50c9de\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="665156e5-83ea-4bf4-a6b7-bba23b60ab71" pod="opendatahub/test-curl-7726-50c9de"
Sep 11 17:39:10.052160 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:10.052125    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-50c9de\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="665156e5-83ea-4bf4-a6b7-bba23b60ab71" pod="opendatahub/test-curl-7726-50c9de"
Sep 11 17:39:10.090567 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.090534    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-08d62b"
Sep 11 17:39:10.093661 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:10.093628    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-08d62b\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="e97df258-1e45-42b2-ae4c-1c91d93ca6cb" pod="opendatahub/test-curl-7726-08d62b"
Sep 11 17:39:10.115531 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.115506    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnjhx\" (UniqueName: \"kubernetes.io/projected/e97df258-1e45-42b2-ae4c-1c91d93ca6cb-kube-api-access-vnjhx\") pod \"e97df258-1e45-42b2-ae4c-1c91d93ca6cb\" (UID: \"e97df258-1e45-42b2-ae4c-1c91d93ca6cb\") "
Sep 11 17:39:10.117826 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.117801    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97df258-1e45-42b2-ae4c-1c91d93ca6cb-kube-api-access-vnjhx" (OuterVolumeSpecName: "kube-api-access-vnjhx") pod "e97df258-1e45-42b2-ae4c-1c91d93ca6cb" (UID: "e97df258-1e45-42b2-ae4c-1c91d93ca6cb"). InnerVolumeSpecName "kube-api-access-vnjhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:39:10.178049 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.177974    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:10.216766 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.216734    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnjhx\" (UniqueName: \"kubernetes.io/projected/e97df258-1e45-42b2-ae4c-1c91d93ca6cb-kube-api-access-vnjhx\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:39:10.545235 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.545144    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7726-4b363e"]
Sep 11 17:39:10.551394 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.551363    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7726-4b363e"]
Sep 11 17:39:10.660567 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.660533    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7726-d548c0"]
Sep 11 17:39:10.660921 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.660909    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30" containerName="test-curl-7726-4b363e"
Sep 11 17:39:10.660969 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.660922    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30" containerName="test-curl-7726-4b363e"
Sep 11 17:39:10.661002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.660981    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30" containerName="test-curl-7726-4b363e"
Sep 11 17:39:10.663065 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.663050    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-d548c0"
Sep 11 17:39:10.665218 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:10.665193    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-08d62b\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="e97df258-1e45-42b2-ae4c-1c91d93ca6cb" pod="opendatahub/test-curl-7726-08d62b"
Sep 11 17:39:10.672713 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.672689    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7726-d548c0"]
Sep 11 17:39:10.721246 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.721212    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl5cs\" (UniqueName: \"kubernetes.io/projected/5810883d-096e-4cbe-89cd-6dd32540535d-kube-api-access-tl5cs\") pod \"test-curl-7726-d548c0\" (UID: \"5810883d-096e-4cbe-89cd-6dd32540535d\") " pod="opendatahub/test-curl-7726-d548c0"
Sep 11 17:39:10.822707 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.822620    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl5cs\" (UniqueName: \"kubernetes.io/projected/5810883d-096e-4cbe-89cd-6dd32540535d-kube-api-access-tl5cs\") pod \"test-curl-7726-d548c0\" (UID: \"5810883d-096e-4cbe-89cd-6dd32540535d\") " pod="opendatahub/test-curl-7726-d548c0"
Sep 11 17:39:10.831126 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.831101    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl5cs\" (UniqueName: \"kubernetes.io/projected/5810883d-096e-4cbe-89cd-6dd32540535d-kube-api-access-tl5cs\") pod \"test-curl-7726-d548c0\" (UID: \"5810883d-096e-4cbe-89cd-6dd32540535d\") " pod="opendatahub/test-curl-7726-d548c0"
Sep 11 17:39:10.912052 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.912020    2560 generic.go:356] "Generic (PLEG): container finished" podID="e97df258-1e45-42b2-ae4c-1c91d93ca6cb" containerID="f8f94ba664fa39442bdab0bf3ec890338d66acd0c9ee873622e6ecb2f99d240b" exitCode=137
Sep 11 17:39:10.912537 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.912062    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-08d62b"
Sep 11 17:39:10.912537 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.912128    2560 scope.go:117] "RemoveContainer" containerID="f8f94ba664fa39442bdab0bf3ec890338d66acd0c9ee873622e6ecb2f99d240b"
Sep 11 17:39:10.912537 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:10.912326    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7726-4b363e" container="test-curl-7726-4b363e"
Sep 11 17:39:10.914809 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:10.914779    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-08d62b\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="e97df258-1e45-42b2-ae4c-1c91d93ca6cb" pod="opendatahub/test-curl-7726-08d62b"
Sep 11 17:39:10.921959 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.921942    2560 scope.go:117] "RemoveContainer" containerID="f8f94ba664fa39442bdab0bf3ec890338d66acd0c9ee873622e6ecb2f99d240b"
Sep 11 17:39:10.922265 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:10.922249    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f94ba664fa39442bdab0bf3ec890338d66acd0c9ee873622e6ecb2f99d240b\": container with ID starting with f8f94ba664fa39442bdab0bf3ec890338d66acd0c9ee873622e6ecb2f99d240b not found: ID does not exist" containerID="f8f94ba664fa39442bdab0bf3ec890338d66acd0c9ee873622e6ecb2f99d240b"
Sep 11 17:39:10.922310 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.922276    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f94ba664fa39442bdab0bf3ec890338d66acd0c9ee873622e6ecb2f99d240b"} err="failed to get container status \"f8f94ba664fa39442bdab0bf3ec890338d66acd0c9ee873622e6ecb2f99d240b\": rpc error: code = NotFound desc = could not find container \"f8f94ba664fa39442bdab0bf3ec890338d66acd0c9ee873622e6ecb2f99d240b\": container with ID starting with f8f94ba664fa39442bdab0bf3ec890338d66acd0c9ee873622e6ecb2f99d240b not found: ID does not exist"
Sep 11 17:39:10.924460 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:10.924437    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-08d62b\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="e97df258-1e45-42b2-ae4c-1c91d93ca6cb" pod="opendatahub/test-curl-7726-08d62b"
Sep 11 17:39:10.972640 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:10.972596    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-d548c0"
Sep 11 17:39:11.048069 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:11.048039    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:11.243143 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:11.243118    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7726-d548c0"]
Sep 11 17:39:11.245180 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:39:11.245145    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5810883d_096e_4cbe_89cd_6dd32540535d.slice/crio-814f166dafa67fbadaddd895a85e0be844ae5c379fcff9d6ce90c793ba5c3132 WatchSource:0}: Error finding container 814f166dafa67fbadaddd895a85e0be844ae5c379fcff9d6ce90c793ba5c3132: Status 404 returned error can't find the container with id 814f166dafa67fbadaddd895a85e0be844ae5c379fcff9d6ce90c793ba5c3132
Sep 11 17:39:11.748428 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:11.748394    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:11.851572 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:11.851531    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:11.921663 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:11.921629    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7726-d548c0" event={"ID":"5810883d-096e-4cbe-89cd-6dd32540535d","Type":"ContainerStarted","Data":"92055c5f584ef86df887251d92034d2e62860438f55490631d715b3111cc8dc4"}
Sep 11 17:39:11.921974 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:11.921678    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7726-d548c0" event={"ID":"5810883d-096e-4cbe-89cd-6dd32540535d","Type":"ContainerStarted","Data":"814f166dafa67fbadaddd895a85e0be844ae5c379fcff9d6ce90c793ba5c3132"}
Sep 11 17:39:11.921974 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:11.921681    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7726-4b363e" podUID="4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30" containerName="test-curl-7726-4b363e" containerID="cri-o://fb6733bdd1739387a87a2df397ae6468baf70ff84421e41bda4e39cac57a0af6" gracePeriod=2
Sep 11 17:39:11.924262 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:11.924232    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-4b363e\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30" pod="opendatahub/test-curl-7726-4b363e"
Sep 11 17:39:11.946306 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:11.946255    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7726-d548c0" podStartSLOduration=1.946240352 podStartE2EDuration="1.946240352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:11.945177181 +0000 UTC m=+1122.377629959" watchObservedRunningTime="2026-09-11 17:39:11.946240352 +0000 UTC m=+1122.378693129"
Sep 11 17:39:11.948345 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:11.948318    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-08d62b\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="e97df258-1e45-42b2-ae4c-1c91d93ca6cb" pod="opendatahub/test-curl-7726-08d62b"
Sep 11 17:39:11.951067 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:11.951043    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-08d62b\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="e97df258-1e45-42b2-ae4c-1c91d93ca6cb" pod="opendatahub/test-curl-7726-08d62b"
Sep 11 17:39:11.952847 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:11.952826    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-4b363e\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30" pod="opendatahub/test-curl-7726-4b363e"
Sep 11 17:39:11.984469 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:11.984436    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:12.051548 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:12.051521    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e97df258-1e45-42b2-ae4c-1c91d93ca6cb" path="/var/lib/kubelet/pods/e97df258-1e45-42b2-ae4c-1c91d93ca6cb/volumes"
Sep 11 17:39:12.098645 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:12.098612    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-50c9de"
Sep 11 17:39:12.136926 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:12.136895    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j84p\" (UniqueName: \"kubernetes.io/projected/665156e5-83ea-4bf4-a6b7-bba23b60ab71-kube-api-access-7j84p\") pod \"665156e5-83ea-4bf4-a6b7-bba23b60ab71\" (UID: \"665156e5-83ea-4bf4-a6b7-bba23b60ab71\") "
Sep 11 17:39:12.139310 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:12.139277    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665156e5-83ea-4bf4-a6b7-bba23b60ab71-kube-api-access-7j84p" (OuterVolumeSpecName: "kube-api-access-7j84p") pod "665156e5-83ea-4bf4-a6b7-bba23b60ab71" (UID: "665156e5-83ea-4bf4-a6b7-bba23b60ab71"). InnerVolumeSpecName "kube-api-access-7j84p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:39:12.238700 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:12.238667    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j84p\" (UniqueName: \"kubernetes.io/projected/665156e5-83ea-4bf4-a6b7-bba23b60ab71-kube-api-access-7j84p\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:39:12.353047 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:12.353013    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:12.456778 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:12.456745    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:12.541295 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:12.541263    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7726-d548c0"]
Sep 11 17:39:12.545453 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:12.545421    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7726-d548c0"]
Sep 11 17:39:12.928324 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:12.928283    2560 generic.go:356] "Generic (PLEG): container finished" podID="665156e5-83ea-4bf4-a6b7-bba23b60ab71" containerID="68cf270dfbcff73eb6ee35870b0fde446c051fe42c93744b9d57afc05da730a2" exitCode=137
Sep 11 17:39:12.928750 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:12.928349    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-50c9de"
Sep 11 17:39:12.928750 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:12.928380    2560 scope.go:117] "RemoveContainer" containerID="68cf270dfbcff73eb6ee35870b0fde446c051fe42c93744b9d57afc05da730a2"
Sep 11 17:39:12.928750 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:12.928658    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7726-d548c0" container="test-curl-7726-d548c0"
Sep 11 17:39:12.939721 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:12.939701    2560 scope.go:117] "RemoveContainer" containerID="68cf270dfbcff73eb6ee35870b0fde446c051fe42c93744b9d57afc05da730a2"
Sep 11 17:39:12.940041 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:12.940021    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cf270dfbcff73eb6ee35870b0fde446c051fe42c93744b9d57afc05da730a2\": container with ID starting with 68cf270dfbcff73eb6ee35870b0fde446c051fe42c93744b9d57afc05da730a2 not found: ID does not exist" containerID="68cf270dfbcff73eb6ee35870b0fde446c051fe42c93744b9d57afc05da730a2"
Sep 11 17:39:12.940104 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:12.940056    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cf270dfbcff73eb6ee35870b0fde446c051fe42c93744b9d57afc05da730a2"} err="failed to get container status \"68cf270dfbcff73eb6ee35870b0fde446c051fe42c93744b9d57afc05da730a2\": rpc error: code = NotFound desc = could not find container \"68cf270dfbcff73eb6ee35870b0fde446c051fe42c93744b9d57afc05da730a2\": container with ID starting with 68cf270dfbcff73eb6ee35870b0fde446c051fe42c93744b9d57afc05da730a2 not found: ID does not exist"
Sep 11 17:39:12.941309 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:12.941284    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-50c9de\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="665156e5-83ea-4bf4-a6b7-bba23b60ab71" pod="opendatahub/test-curl-7726-50c9de"
Sep 11 17:39:13.128359 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.128327    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:13.402529 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.402491    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"]
Sep 11 17:39:13.403118 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.403096    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5810883d-096e-4cbe-89cd-6dd32540535d" containerName="test-curl-7726-d548c0"
Sep 11 17:39:13.403163 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.403123    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5810883d-096e-4cbe-89cd-6dd32540535d" containerName="test-curl-7726-d548c0"
Sep 11 17:39:13.403270 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.403257    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5810883d-096e-4cbe-89cd-6dd32540535d" containerName="test-curl-7726-d548c0"
Sep 11 17:39:13.406226 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.406197    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.409291 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.409272    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3d144889-openshift-default-dockercfg-g699n\""
Sep 11 17:39:13.419052 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.419027    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"]
Sep 11 17:39:13.432982 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.432959    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:13.443229 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:13.443203    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-50c9de\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="665156e5-83ea-4bf4-a6b7-bba23b60ab71" pod="opendatahub/test-curl-7726-50c9de"
Sep 11 17:39:13.452839 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.452815    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2855f490-c549-4dd0-b96c-7f7c82a67118-istiod-ca-cert\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.452952 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.452851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-token\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.452952 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.452869    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-data\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.452952 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.452895    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-workload-certs\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.453058 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.452974    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-podinfo\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.453058 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.453018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs4wv\" (UniqueName: \"kubernetes.io/projected/2855f490-c549-4dd0-b96c-7f7c82a67118-kube-api-access-xs4wv\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.453122 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.453062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-credential-socket\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.453122 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.453084    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-envoy\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.453187 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.453154    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-workload-socket\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.554407 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.554355    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-podinfo\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.554583 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.554430    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs4wv\" (UniqueName: \"kubernetes.io/projected/2855f490-c549-4dd0-b96c-7f7c82a67118-kube-api-access-xs4wv\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.554583 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.554498    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-credential-socket\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.554583 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.554531    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-envoy\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.554759 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.554698    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-workload-socket\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.554815 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.554772    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2855f490-c549-4dd0-b96c-7f7c82a67118-istiod-ca-cert\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.554870 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.554816    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-token\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.554870 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.554850    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-data\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.554974 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.554899    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-workload-certs\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.555023 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.554987    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-credential-socket\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.555077 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.555058    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-workload-socket\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.555286 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.555253    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-workload-certs\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.555415 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.555290    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-data\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.555646 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.555624    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2855f490-c549-4dd0-b96c-7f7c82a67118-istiod-ca-cert\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.556797 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.556779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-envoy\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.557102 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.557087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-podinfo\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.566089 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.566039    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-token\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.567120 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.567098    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs4wv\" (UniqueName: \"kubernetes.io/projected/2855f490-c549-4dd0-b96c-7f7c82a67118-kube-api-access-xs4wv\") pod \"e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") " pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.622084 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.622055    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:13.717649 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.717556    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:13.934432 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.934399    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7726-d548c0" podUID="5810883d-096e-4cbe-89cd-6dd32540535d" containerName="test-curl-7726-d548c0" containerID="cri-o://92055c5f584ef86df887251d92034d2e62860438f55490631d715b3111cc8dc4" gracePeriod=2
Sep 11 17:39:13.936544 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:13.936499    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-d548c0\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="5810883d-096e-4cbe-89cd-6dd32540535d" pod="opendatahub/test-curl-7726-d548c0"
Sep 11 17:39:13.938359 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:13.938335    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-50c9de\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="665156e5-83ea-4bf4-a6b7-bba23b60ab71" pod="opendatahub/test-curl-7726-50c9de"
Sep 11 17:39:13.992129 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.992098    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"]
Sep 11 17:39:13.994510 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:39:13.994468    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2855f490_c549_4dd0_b96c_7f7c82a67118.slice/crio-76c6d683ff73157dc4b4e00009b12a8a9549e65ed682f233a61d4fda81ee5f75 WatchSource:0}: Error finding container 76c6d683ff73157dc4b4e00009b12a8a9549e65ed682f233a61d4fda81ee5f75: Status 404 returned error can't find the container with id 76c6d683ff73157dc4b4e00009b12a8a9549e65ed682f233a61d4fda81ee5f75
Sep 11 17:39:13.996861 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.996830    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:39:13.996935 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.996895    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:39:13.996935 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:13.996922    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:39:14.053337 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:14.053227    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="665156e5-83ea-4bf4-a6b7-bba23b60ab71" path="/var/lib/kubelet/pods/665156e5-83ea-4bf4-a6b7-bba23b60ab71/volumes"
Sep 11 17:39:14.093835 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:14.093813    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-4b363e"
Sep 11 17:39:14.160758 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:14.160727    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmz8b\" (UniqueName: \"kubernetes.io/projected/4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30-kube-api-access-pmz8b\") pod \"4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30\" (UID: \"4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30\") "
Sep 11 17:39:14.162797 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:14.162770    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30-kube-api-access-pmz8b" (OuterVolumeSpecName: "kube-api-access-pmz8b") pod "4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30" (UID: "4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30"). InnerVolumeSpecName "kube-api-access-pmz8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:39:14.262117 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:14.262032    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmz8b\" (UniqueName: \"kubernetes.io/projected/4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30-kube-api-access-pmz8b\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:39:14.940623 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:14.940583    2560 generic.go:356] "Generic (PLEG): container finished" podID="4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30" containerID="fb6733bdd1739387a87a2df397ae6468baf70ff84421e41bda4e39cac57a0af6" exitCode=137
Sep 11 17:39:14.940623 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:14.940621    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-4b363e"
Sep 11 17:39:14.941107 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:14.940686    2560 scope.go:117] "RemoveContainer" containerID="fb6733bdd1739387a87a2df397ae6468baf70ff84421e41bda4e39cac57a0af6"
Sep 11 17:39:14.942645 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:14.942613    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl" event={"ID":"2855f490-c549-4dd0-b96c-7f7c82a67118","Type":"ContainerStarted","Data":"7e675439449f2df5a80f3c21e7d081dc260f381d05519b0be1c5f6570ffa92e9"}
Sep 11 17:39:14.942763 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:14.942674    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl" event={"ID":"2855f490-c549-4dd0-b96c-7f7c82a67118","Type":"ContainerStarted","Data":"76c6d683ff73157dc4b4e00009b12a8a9549e65ed682f233a61d4fda81ee5f75"}
Sep 11 17:39:14.949867 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:14.949849    2560 scope.go:117] "RemoveContainer" containerID="fb6733bdd1739387a87a2df397ae6468baf70ff84421e41bda4e39cac57a0af6"
Sep 11 17:39:14.950139 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:14.950117    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6733bdd1739387a87a2df397ae6468baf70ff84421e41bda4e39cac57a0af6\": container with ID starting with fb6733bdd1739387a87a2df397ae6468baf70ff84421e41bda4e39cac57a0af6 not found: ID does not exist" containerID="fb6733bdd1739387a87a2df397ae6468baf70ff84421e41bda4e39cac57a0af6"
Sep 11 17:39:14.950200 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:14.950151    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6733bdd1739387a87a2df397ae6468baf70ff84421e41bda4e39cac57a0af6"} err="failed to get container status \"fb6733bdd1739387a87a2df397ae6468baf70ff84421e41bda4e39cac57a0af6\": rpc error: code = NotFound desc = could not find container \"fb6733bdd1739387a87a2df397ae6468baf70ff84421e41bda4e39cac57a0af6\": container with ID starting with fb6733bdd1739387a87a2df397ae6468baf70ff84421e41bda4e39cac57a0af6 not found: ID does not exist"
Sep 11 17:39:14.965653 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:14.965605    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl" podStartSLOduration=1.965588356 podStartE2EDuration="1.965588356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:14.963971374 +0000 UTC m=+1125.396424151" watchObservedRunningTime="2026-09-11 17:39:14.965588356 +0000 UTC m=+1125.398041202"
Sep 11 17:39:14.966650 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:14.966627    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-4b363e\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30" pod="opendatahub/test-curl-7726-4b363e"
Sep 11 17:39:15.718086 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:15.718035    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:15.723191 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:15.723160    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:15.725521 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:15.725467    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-4b363e\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30" pod="opendatahub/test-curl-7726-4b363e"
Sep 11 17:39:15.949847 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:15.949821    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:15.950871 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:15.950851    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:15.972544 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:15.972501    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-4b363e\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30" pod="opendatahub/test-curl-7726-4b363e"
Sep 11 17:39:16.053593 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:16.053563    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30" path="/var/lib/kubelet/pods/4716bc1a-8eb8-489b-8ca9-fdb5c68dbc30/volumes"
Sep 11 17:39:16.116202 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:16.116175    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-d548c0"
Sep 11 17:39:16.141056 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:16.141026    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:16.186858 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:16.186818    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl5cs\" (UniqueName: \"kubernetes.io/projected/5810883d-096e-4cbe-89cd-6dd32540535d-kube-api-access-tl5cs\") pod \"5810883d-096e-4cbe-89cd-6dd32540535d\" (UID: \"5810883d-096e-4cbe-89cd-6dd32540535d\") "
Sep 11 17:39:16.188918 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:16.188892    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5810883d-096e-4cbe-89cd-6dd32540535d-kube-api-access-tl5cs" (OuterVolumeSpecName: "kube-api-access-tl5cs") pod "5810883d-096e-4cbe-89cd-6dd32540535d" (UID: "5810883d-096e-4cbe-89cd-6dd32540535d"). InnerVolumeSpecName "kube-api-access-tl5cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:39:16.288468 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:16.288375    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl5cs\" (UniqueName: \"kubernetes.io/projected/5810883d-096e-4cbe-89cd-6dd32540535d-kube-api-access-tl5cs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:39:16.948815 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:16.948782    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:16.956408 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:16.956380    2560 generic.go:356] "Generic (PLEG): container finished" podID="5810883d-096e-4cbe-89cd-6dd32540535d" containerID="92055c5f584ef86df887251d92034d2e62860438f55490631d715b3111cc8dc4" exitCode=137
Sep 11 17:39:16.956833 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:16.956426    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7726-d548c0"
Sep 11 17:39:16.956833 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:16.956487    2560 scope.go:117] "RemoveContainer" containerID="92055c5f584ef86df887251d92034d2e62860438f55490631d715b3111cc8dc4"
Sep 11 17:39:16.965872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:16.965852    2560 scope.go:117] "RemoveContainer" containerID="92055c5f584ef86df887251d92034d2e62860438f55490631d715b3111cc8dc4"
Sep 11 17:39:16.966150 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:16.966133    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92055c5f584ef86df887251d92034d2e62860438f55490631d715b3111cc8dc4\": container with ID starting with 92055c5f584ef86df887251d92034d2e62860438f55490631d715b3111cc8dc4 not found: ID does not exist" containerID="92055c5f584ef86df887251d92034d2e62860438f55490631d715b3111cc8dc4"
Sep 11 17:39:16.966198 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:16.966158    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92055c5f584ef86df887251d92034d2e62860438f55490631d715b3111cc8dc4"} err="failed to get container status \"92055c5f584ef86df887251d92034d2e62860438f55490631d715b3111cc8dc4\": rpc error: code = NotFound desc = could not find container \"92055c5f584ef86df887251d92034d2e62860438f55490631d715b3111cc8dc4\": container with ID starting with 92055c5f584ef86df887251d92034d2e62860438f55490631d715b3111cc8dc4 not found: ID does not exist"
Sep 11 17:39:16.967502 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:39:16.967465    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7726-d548c0\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="5810883d-096e-4cbe-89cd-6dd32540535d" pod="opendatahub/test-curl-7726-d548c0"
Sep 11 17:39:17.060318 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:17.060280    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:17.848707 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:17.848666    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:18.051874 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:18.051838    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5810883d-096e-4cbe-89cd-6dd32540535d" path="/var/lib/kubelet/pods/5810883d-096e-4cbe-89cd-6dd32540535d/volumes"
Sep 11 17:39:18.839864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:18.839830    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:19.654757 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:19.654721    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:20.436141 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:20.436110    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:20.538019 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:20.537979    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:20.987676 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:20.987629    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:21.078758 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:21.078726    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:21.147986 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:21.147922    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:29.044262 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:29.044226    2560 generic.go:356] "Generic (PLEG): container finished" podID="ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" containerID="510e99a01ed52d6c29cae813d2ff0ce2d5ee416a0a4364e014eaf108a7954e2a" exitCode=0
Sep 11 17:39:29.044834 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:29.044305    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx" event={"ID":"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202","Type":"ContainerDied","Data":"510e99a01ed52d6c29cae813d2ff0ce2d5ee416a0a4364e014eaf108a7954e2a"}
Sep 11 17:39:29.044903 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:29.044891    2560 scope.go:117] "RemoveContainer" containerID="510e99a01ed52d6c29cae813d2ff0ce2d5ee416a0a4364e014eaf108a7954e2a"
Sep 11 17:39:29.046002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:29.045975    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:39:29.046066 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:29.046039    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:39:29.046120 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:29.046072    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:39:30.061734 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:30.061702    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx" event={"ID":"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202","Type":"ContainerStarted","Data":"b5586f314324ab2aa957f7bc43b9155da3eb38d003704081edcf9feabfed3127"}
Sep 11 17:39:30.139524 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:30.139463    2560 scope.go:117] "RemoveContainer" containerID="a96661fc86d12c0eedba843769766bb2f2f16aaa2eee3a0a415b8c5d06cf97a4"
Sep 11 17:39:30.447869 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:30.447822    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:30.448243 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:30.447958    2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 11 17:39:30.448436 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:30.448409    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx" podUID="ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 11 17:39:31.447716 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:31.447666    2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 11 17:39:31.448307 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:31.447739    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx" podUID="ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 11 17:39:32.447693 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:32.447645    2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 11 17:39:32.447993 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:32.447961    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx" podUID="ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 11 17:39:33.447310 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:33.447267    2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 11 17:39:33.447551 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:33.447328    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx" podUID="ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 11 17:39:33.770083 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:33.769990    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:34.447042 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:34.447003    2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 11 17:39:34.447218 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:34.447064    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx" podUID="ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 11 17:39:35.202154 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:35.202120    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:35.452783 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:35.452700    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:35.452997 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:35.452977    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:35.453742 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:35.453724    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:39:35.947632 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:35.947595    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:36.675439 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:36.675403    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:36.775647 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:36.775607    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:36.920210 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:36.920175    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:37.101378 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:37.101296    2560 generic.go:356] "Generic (PLEG): container finished" podID="2855f490-c549-4dd0-b96c-7f7c82a67118" containerID="7e675439449f2df5a80f3c21e7d081dc260f381d05519b0be1c5f6570ffa92e9" exitCode=0
Sep 11 17:39:37.101568 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:37.101372    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl" event={"ID":"2855f490-c549-4dd0-b96c-7f7c82a67118","Type":"ContainerDied","Data":"7e675439449f2df5a80f3c21e7d081dc260f381d05519b0be1c5f6570ffa92e9"}
Sep 11 17:39:37.101925 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:37.101910    2560 scope.go:117] "RemoveContainer" containerID="7e675439449f2df5a80f3c21e7d081dc260f381d05519b0be1c5f6570ffa92e9"
Sep 11 17:39:37.103067 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:37.103042    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:39:37.103126 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:37.103110    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:39:37.103163 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:37.103139    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:39:37.465158 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:37.465128    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:38.110497 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:38.110428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl" event={"ID":"2855f490-c549-4dd0-b96c-7f7c82a67118","Type":"ContainerStarted","Data":"4323db9f6fe96c746e6d2e8036e13aa0aca23cacb121e67e0424a50aaba867df"}
Sep 11 17:39:38.243438 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:38.243410    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:38.718378 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:38.718273    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:38.718614 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:38.718380    2560 patch_prober.go:28] interesting pod/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 11 17:39:38.718614 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:38.718434    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl" podUID="2855f490-c549-4dd0-b96c-7f7c82a67118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 11 17:39:38.876589 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:38.876556    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:38.975328 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:38.975241    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:39.331535 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.331421    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"]
Sep 11 17:39:39.338792 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.338023    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.353181 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.353143    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3d144889\"/\"openshift-service-ca.crt\""
Sep 11 17:39:39.354545 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.354425    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3d144889\"/\"kube-root-ca.crt\""
Sep 11 17:39:39.354687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.354636    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3d144889\"/\"e2e-lifecycle-model-3d144889-kserve-self-signed-certs\""
Sep 11 17:39:39.354979 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.354959    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3d144889\"/\"default-dockercfg-rfbbb\""
Sep 11 17:39:39.358169 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.358070    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"]
Sep 11 17:39:39.358354 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.358330    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:39.448396 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.448360    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-tmp-dir\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.448608 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.448424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-tls-certs\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.448608 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.448541    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-model-cache\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.448685 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.448609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hln9b\" (UniqueName: \"kubernetes.io/projected/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-kube-api-access-hln9b\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.448685 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.448680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-home\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.448761 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.448703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-dshm\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.549639 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.549598    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-model-cache\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.549826 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.549666    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hln9b\" (UniqueName: \"kubernetes.io/projected/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-kube-api-access-hln9b\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.549826 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.549713    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-home\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.549826 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.549730    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-dshm\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.549999 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.549870    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-tmp-dir\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.549999 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.549967    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-tls-certs\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.550145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.550123    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-model-cache\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.550184 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.550140    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-home\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.550230 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.550210    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-tmp-dir\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.551885 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.551859    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-dshm\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.552248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.552231    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-tls-certs\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.558381 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.558364    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hln9b\" (UniqueName: \"kubernetes.io/projected/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-kube-api-access-hln9b\") pod \"e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") " pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.644609 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.644570    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:39.660567 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.660533    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:39.723501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.722607    2560 patch_prober.go:28] interesting pod/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 11 17:39:39.723501 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.722693    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl" podUID="2855f490-c549-4dd0-b96c-7f7c82a67118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 11 17:39:39.966924 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:39.966898    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"]
Sep 11 17:39:39.968719 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:39:39.968681    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07df38fd_c576_4711_a6bb_8bb0e9adbc9d.slice/crio-5f3b701ff1b7090951fe6a7f6b563a63a41f7cffa3d5c4989f4d36bca3e79bf3 WatchSource:0}: Error finding container 5f3b701ff1b7090951fe6a7f6b563a63a41f7cffa3d5c4989f4d36bca3e79bf3: Status 404 returned error can't find the container with id 5f3b701ff1b7090951fe6a7f6b563a63a41f7cffa3d5c4989f4d36bca3e79bf3
Sep 11 17:39:40.131305 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:40.131260    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g" event={"ID":"07df38fd-c576-4711-a6bb-8bb0e9adbc9d","Type":"ContainerStarted","Data":"d5b3b9f05a4b77a34c70195b36c715b50c07d082dd5be150718de858a4a5c565"}
Sep 11 17:39:40.131305 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:40.131302    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g" event={"ID":"07df38fd-c576-4711-a6bb-8bb0e9adbc9d","Type":"ContainerStarted","Data":"5f3b701ff1b7090951fe6a7f6b563a63a41f7cffa3d5c4989f4d36bca3e79bf3"}
Sep 11 17:39:40.149017 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:40.148953    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g" podStartSLOduration=1.148931168 podStartE2EDuration="1.148931168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:40.146950824 +0000 UTC m=+1150.579403603" watchObservedRunningTime="2026-09-11 17:39:40.148931168 +0000 UTC m=+1150.581383946"
Sep 11 17:39:40.422283 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:40.422253    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:40.718085 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:40.717980    2560 patch_prober.go:28] interesting pod/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 11 17:39:40.718085 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:40.718054    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl" podUID="2855f490-c549-4dd0-b96c-7f7c82a67118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 11 17:39:41.330231 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:41.330181    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:41.718917 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:41.718873    2560 patch_prober.go:28] interesting pod/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 11 17:39:41.719530 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:41.719468    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl" podUID="2855f490-c549-4dd0-b96c-7f7c82a67118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 11 17:39:42.719057 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.719018    2560 patch_prober.go:28] interesting pod/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 11 17:39:42.719706 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.719662    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl" podUID="2855f490-c549-4dd0-b96c-7f7c82a67118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 11 17:39:42.731134 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.731089    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"]
Sep 11 17:39:42.738468 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.738443    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.745514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.744626    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4c36f4\"/\"default-dockercfg-s747w\""
Sep 11 17:39:42.745514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.744684    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4c36f4\"/\"kube-root-ca.crt\""
Sep 11 17:39:42.745514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.744626    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4c36f4\"/\"auth-test-model-4c36f4-kserve-self-signed-certs\""
Sep 11 17:39:42.745514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.744923    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4c36f4\"/\"openshift-service-ca.crt\""
Sep 11 17:39:42.746820 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.746778    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"]
Sep 11 17:39:42.767775 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.767739    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:39:42.784181 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.784147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-tls-certs\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.784350 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.784207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-tmp-dir\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.784350 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.784290    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-model-cache\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.784350 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.784324    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-home\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.784459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.784366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-dshm\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.784459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.784391    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h64g\" (UniqueName: \"kubernetes.io/projected/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-kube-api-access-7h64g\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.885217 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.885167    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-tls-certs\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.885415 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.885238    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-tmp-dir\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.885415 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.885279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-model-cache\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.885415 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.885305    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-home\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.885415 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.885352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-dshm\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.885415 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.885368    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h64g\" (UniqueName: \"kubernetes.io/projected/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-kube-api-access-7h64g\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.885865 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.885841    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-model-cache\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.885963 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.885921    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-home\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.886032 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.885981    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-tmp-dir\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.887831 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.887798    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-dshm\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.888074 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.888058    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-tls-certs\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:42.894149 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:42.894124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h64g\" (UniqueName: \"kubernetes.io/projected/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-kube-api-access-7h64g\") pod \"auth-test-model-4c36f4-kserve-8468dff475-58dwb\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:43.053378 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:43.053271    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:43.332458 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:43.332430    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"]
Sep 11 17:39:43.333372 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:39:43.333347    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54cbbc5f_92a1_447d_b739_4ccbfccf6d43.slice/crio-848d92abf5a31245145794e02d6f3bc9a99e2cad02ac997fdffcdbdabb481757 WatchSource:0}: Error finding container 848d92abf5a31245145794e02d6f3bc9a99e2cad02ac997fdffcdbdabb481757: Status 404 returned error can't find the container with id 848d92abf5a31245145794e02d6f3bc9a99e2cad02ac997fdffcdbdabb481757
Sep 11 17:39:43.722412 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:43.722384    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:43.722852 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:43.722704    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:43.723365 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:43.723348    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:39:44.157610 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:44.157578    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb" event={"ID":"54cbbc5f-92a1-447d-b739-4ccbfccf6d43","Type":"ContainerStarted","Data":"a1778a6d59cd586cf3bd74858380c0a869f02ec9bd7b2641db6a33d325a8fc59"}
Sep 11 17:39:44.157792 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:44.157622    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb" event={"ID":"54cbbc5f-92a1-447d-b739-4ccbfccf6d43","Type":"ContainerStarted","Data":"848d92abf5a31245145794e02d6f3bc9a99e2cad02ac997fdffcdbdabb481757"}
Sep 11 17:39:44.191216 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:44.191169    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb" podStartSLOduration=2.191151699 podStartE2EDuration="2.191151699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:39:44.189891829 +0000 UTC m=+1154.622344606" watchObservedRunningTime="2026-09-11 17:39:44.191151699 +0000 UTC m=+1154.623604475"
Sep 11 17:39:49.660900 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:49.660863    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:49.661400 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:49.660918    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:49.673595 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:49.673571    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:50.200638 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:50.200608    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:39:53.053619 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:53.053538    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:53.053619 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:53.053585    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:53.066439 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:53.066411    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:39:53.216529 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:39:53.216500    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:40:06.218990 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:06.218938    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:11.003595 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:11.003561    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv"]
Sep 11 17:40:11.006170 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:11.006154    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv"
Sep 11 17:40:11.008774 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:11.008750    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:40:11.008883 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:11.008750    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:40:11.014305 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:11.014275    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv"]
Sep 11 17:40:11.072921 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:11.072882    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8b33bf-9784-4f79-9e11-1eff31b5ffac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d144889-kjjzv\" (UID: \"fe8b33bf-9784-4f79-9e11-1eff31b5ffac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv"
Sep 11 17:40:11.174297 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:11.174263    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8b33bf-9784-4f79-9e11-1eff31b5ffac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d144889-kjjzv\" (UID: \"fe8b33bf-9784-4f79-9e11-1eff31b5ffac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv"
Sep 11 17:40:11.174878 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:11.174859    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8b33bf-9784-4f79-9e11-1eff31b5ffac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d144889-kjjzv\" (UID: \"fe8b33bf-9784-4f79-9e11-1eff31b5ffac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv"
Sep 11 17:40:11.318808 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:11.318717    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv"
Sep 11 17:40:11.339662 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:11.339609    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:11.588956 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:11.588878    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv"]
Sep 11 17:40:11.592687 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:40:11.592659    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8b33bf_9784_4f79_9e11_1eff31b5ffac.slice/crio-cfcd4570ebec60324979a05adb095cf0b688de49e79e2322efe5e0dc3ba4a185 WatchSource:0}: Error finding container cfcd4570ebec60324979a05adb095cf0b688de49e79e2322efe5e0dc3ba4a185: Status 404 returned error can't find the container with id cfcd4570ebec60324979a05adb095cf0b688de49e79e2322efe5e0dc3ba4a185
Sep 11 17:40:12.307144 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:12.307045    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv" event={"ID":"fe8b33bf-9784-4f79-9e11-1eff31b5ffac","Type":"ContainerStarted","Data":"ecc1bcc1cbe1704db5cec96083019a69ed651ccfa59a02fa77a16deb9095137e"}
Sep 11 17:40:12.307144 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:12.307103    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv" event={"ID":"fe8b33bf-9784-4f79-9e11-1eff31b5ffac","Type":"ContainerStarted","Data":"cfcd4570ebec60324979a05adb095cf0b688de49e79e2322efe5e0dc3ba4a185"}
Sep 11 17:40:12.328813 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:12.328754    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv" podStartSLOduration=2.328736809 podStartE2EDuration="2.328736809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:12.325734576 +0000 UTC m=+1182.758187352" watchObservedRunningTime="2026-09-11 17:40:12.328736809 +0000 UTC m=+1182.761189585"
Sep 11 17:40:12.335753 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:12.335723    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:13.144071 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:13.144033    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:13.314349 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:13.314317    2560 generic.go:356] "Generic (PLEG): container finished" podID="fe8b33bf-9784-4f79-9e11-1eff31b5ffac" containerID="ecc1bcc1cbe1704db5cec96083019a69ed651ccfa59a02fa77a16deb9095137e" exitCode=0
Sep 11 17:40:13.314754 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:13.314394    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv" event={"ID":"fe8b33bf-9784-4f79-9e11-1eff31b5ffac","Type":"ContainerDied","Data":"ecc1bcc1cbe1704db5cec96083019a69ed651ccfa59a02fa77a16deb9095137e"}
Sep 11 17:40:13.411359 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:13.411280    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:14.056621 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:14.056591    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:14.468915 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:14.468890    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv"
Sep 11 17:40:14.510796 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:14.510766    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8b33bf-9784-4f79-9e11-1eff31b5ffac-maas-api-service-ca\") pod \"fe8b33bf-9784-4f79-9e11-1eff31b5ffac\" (UID: \"fe8b33bf-9784-4f79-9e11-1eff31b5ffac\") "
Sep 11 17:40:14.511116 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:14.511094    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8b33bf-9784-4f79-9e11-1eff31b5ffac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe8b33bf-9784-4f79-9e11-1eff31b5ffac" (UID: "fe8b33bf-9784-4f79-9e11-1eff31b5ffac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:40:14.612055 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:14.611963    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe8b33bf-9784-4f79-9e11-1eff31b5ffac-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:40:14.730145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:14.730113    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:15.326392 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:15.326357    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv" event={"ID":"fe8b33bf-9784-4f79-9e11-1eff31b5ffac","Type":"ContainerDied","Data":"cfcd4570ebec60324979a05adb095cf0b688de49e79e2322efe5e0dc3ba4a185"}
Sep 11 17:40:15.326600 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:15.326407    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfcd4570ebec60324979a05adb095cf0b688de49e79e2322efe5e0dc3ba4a185"
Sep 11 17:40:15.326600 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:15.326374    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv"
Sep 11 17:40:15.383983 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:15.383952    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:16.086168 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:16.086098    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:22.430889 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.430854    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"]
Sep 11 17:40:22.431441 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.431420    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe8b33bf-9784-4f79-9e11-1eff31b5ffac" containerName="revoke-keys"
Sep 11 17:40:22.431539 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.431444    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe8b33bf-9784-4f79-9e11-1eff31b5ffac" containerName="revoke-keys"
Sep 11 17:40:22.431605 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.431577    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe8b33bf-9784-4f79-9e11-1eff31b5ffac" containerName="revoke-keys"
Sep 11 17:40:22.433728 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.433707    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.436736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.436713    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-db95fd\"/\"openshift-service-ca.crt\""
Sep 11 17:40:22.436881 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.436758    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-db95fd\"/\"default-dockercfg-fhfrq\""
Sep 11 17:40:22.439002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.438983    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-db95fd\"/\"kube-root-ca.crt\""
Sep 11 17:40:22.439223 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.439201    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-db95fd\"/\"auth-test-model-db95fd-kserve-self-signed-certs\""
Sep 11 17:40:22.443777 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.443756    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"]
Sep 11 17:40:22.454065 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.454038    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:22.495913 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.495885    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-model-cache\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.496043 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.495928    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-tmp-dir\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.496043 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.495969    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxtcc\" (UniqueName: \"kubernetes.io/projected/68160d51-663c-4b0e-80d1-c9c924712f12-kube-api-access-jxtcc\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.496043 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.496005    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-home\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.496043 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.496030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-dshm\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.496251 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.496147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68160d51-663c-4b0e-80d1-c9c924712f12-tls-certs\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.597088 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.597047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-model-cache\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.597289 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.597110    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-tmp-dir\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.597289 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.597155    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxtcc\" (UniqueName: \"kubernetes.io/projected/68160d51-663c-4b0e-80d1-c9c924712f12-kube-api-access-jxtcc\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.597289 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.597203    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-home\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.597289 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.597240    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-dshm\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.597531 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.597325    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68160d51-663c-4b0e-80d1-c9c924712f12-tls-certs\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.597590 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.597544    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-model-cache\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.599447 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.598052    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-tmp-dir\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.599447 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.598087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-home\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.604520 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.600860    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-dshm\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.608295 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.608264    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxtcc\" (UniqueName: \"kubernetes.io/projected/68160d51-663c-4b0e-80d1-c9c924712f12-kube-api-access-jxtcc\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.608673 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.608652    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68160d51-663c-4b0e-80d1-c9c924712f12-tls-certs\") pod \"auth-test-model-db95fd-kserve-797646cd75-t4v2j\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:22.745532 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:22.745429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:23.017678 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:23.017652    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"]
Sep 11 17:40:23.019512 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:40:23.019485    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68160d51_663c_4b0e_80d1_c9c924712f12.slice/crio-b0e519b84fd346ad20b1f832eb1dd3e0b93f0afc8366d227058f1ffe2daca16e WatchSource:0}: Error finding container b0e519b84fd346ad20b1f832eb1dd3e0b93f0afc8366d227058f1ffe2daca16e: Status 404 returned error can't find the container with id b0e519b84fd346ad20b1f832eb1dd3e0b93f0afc8366d227058f1ffe2daca16e
Sep 11 17:40:23.375011 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:23.374891    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j" event={"ID":"68160d51-663c-4b0e-80d1-c9c924712f12","Type":"ContainerStarted","Data":"a97987898ea58088f9246c40c4f262fad14802ad4c03b1bd15d7f681d30ba4af"}
Sep 11 17:40:23.375011 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:23.374953    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j" event={"ID":"68160d51-663c-4b0e-80d1-c9c924712f12","Type":"ContainerStarted","Data":"b0e519b84fd346ad20b1f832eb1dd3e0b93f0afc8366d227058f1ffe2daca16e"}
Sep 11 17:40:23.394162 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:23.394116    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j" podStartSLOduration=1.394102677 podStartE2EDuration="1.394102677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:40:23.392440601 +0000 UTC m=+1193.824893379" watchObservedRunningTime="2026-09-11 17:40:23.394102677 +0000 UTC m=+1193.826555453"
Sep 11 17:40:26.678191 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:26.678158    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:26.748267 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:26.748236    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:26.847961 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:26.847932    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:26.879152 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:26.879113    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv"]
Sep 11 17:40:26.883415 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:26.883391    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d144889-kjjzv"]
Sep 11 17:40:26.924719 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:26.924690    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:27.328850 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:27.328815    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:28.051678 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:28.051642    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe8b33bf-9784-4f79-9e11-1eff31b5ffac" path="/var/lib/kubelet/pods/fe8b33bf-9784-4f79-9e11-1eff31b5ffac/volumes"
Sep 11 17:40:28.158539 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:28.158500    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:29.041359 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:29.041332    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:29.761002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:29.760945    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:30.066079 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:30.065990    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:40:30.068886 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:30.068864    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:40:30.123238 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:30.123197    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:30.189323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:30.189293    2560 scope.go:117] "RemoveContainer" containerID="c7e3684650a80ff7c9331ec6d7cb9e74df458dac16ff3ad02d25c9ac001c8dd3"
Sep 11 17:40:30.429792 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:30.429758    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:32.745689 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:32.745640    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:32.745689 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:32.745703    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:32.758295 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:32.758268    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:33.446119 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:33.446087    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:40:35.097985 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:35.097950    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"]
Sep 11 17:40:35.148085 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:35.148053    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:40.235255 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:40.235218    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:40:50.117845 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.117775    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g" podUID="07df38fd-c576-4711-a6bb-8bb0e9adbc9d" containerName="main" containerID="cri-o://d5b3b9f05a4b77a34c70195b36c715b50c07d082dd5be150718de858a4a5c565" gracePeriod=45
Sep 11 17:40:50.190928 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.190881    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g" podUID="07df38fd-c576-4711-a6bb-8bb0e9adbc9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 11 17:40:50.371524 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.371456    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:40:50.494536 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.494489    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-home\") pod \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") "
Sep 11 17:40:50.494536 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.494542    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-tmp-dir\") pod \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") "
Sep 11 17:40:50.494809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.494561    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-tls-certs\") pod \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") "
Sep 11 17:40:50.494809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.494588    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-model-cache\") pod \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") "
Sep 11 17:40:50.494809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.494669    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-dshm\") pod \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") "
Sep 11 17:40:50.494809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.494694    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hln9b\" (UniqueName: \"kubernetes.io/projected/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-kube-api-access-hln9b\") pod \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\" (UID: \"07df38fd-c576-4711-a6bb-8bb0e9adbc9d\") "
Sep 11 17:40:50.494809 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.494778    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-home" (OuterVolumeSpecName: "home") pod "07df38fd-c576-4711-a6bb-8bb0e9adbc9d" (UID: "07df38fd-c576-4711-a6bb-8bb0e9adbc9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:40:50.495070 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.494856    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07df38fd-c576-4711-a6bb-8bb0e9adbc9d" (UID: "07df38fd-c576-4711-a6bb-8bb0e9adbc9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:40:50.495070 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.494931    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-model-cache" (OuterVolumeSpecName: "model-cache") pod "07df38fd-c576-4711-a6bb-8bb0e9adbc9d" (UID: "07df38fd-c576-4711-a6bb-8bb0e9adbc9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:40:50.495179 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.495103    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:40:50.495179 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.495124    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:40:50.495179 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.495164    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:40:50.497032 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.497002    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-dshm" (OuterVolumeSpecName: "dshm") pod "07df38fd-c576-4711-a6bb-8bb0e9adbc9d" (UID: "07df38fd-c576-4711-a6bb-8bb0e9adbc9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:40:50.497122 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.497011    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-kube-api-access-hln9b" (OuterVolumeSpecName: "kube-api-access-hln9b") pod "07df38fd-c576-4711-a6bb-8bb0e9adbc9d" (UID: "07df38fd-c576-4711-a6bb-8bb0e9adbc9d"). InnerVolumeSpecName "kube-api-access-hln9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:40:50.497122 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.497071    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07df38fd-c576-4711-a6bb-8bb0e9adbc9d" (UID: "07df38fd-c576-4711-a6bb-8bb0e9adbc9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:40:50.527301 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.527267    2560 generic.go:356] "Generic (PLEG): container finished" podID="07df38fd-c576-4711-a6bb-8bb0e9adbc9d" containerID="d5b3b9f05a4b77a34c70195b36c715b50c07d082dd5be150718de858a4a5c565" exitCode=0
Sep 11 17:40:50.527465 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.527308    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"
Sep 11 17:40:50.527465 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.527348    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g" event={"ID":"07df38fd-c576-4711-a6bb-8bb0e9adbc9d","Type":"ContainerDied","Data":"d5b3b9f05a4b77a34c70195b36c715b50c07d082dd5be150718de858a4a5c565"}
Sep 11 17:40:50.527465 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.527391    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g" event={"ID":"07df38fd-c576-4711-a6bb-8bb0e9adbc9d","Type":"ContainerDied","Data":"5f3b701ff1b7090951fe6a7f6b563a63a41f7cffa3d5c4989f4d36bca3e79bf3"}
Sep 11 17:40:50.527465 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.527408    2560 scope.go:117] "RemoveContainer" containerID="d5b3b9f05a4b77a34c70195b36c715b50c07d082dd5be150718de858a4a5c565"
Sep 11 17:40:50.537407 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.537385    2560 scope.go:117] "RemoveContainer" containerID="d5b3b9f05a4b77a34c70195b36c715b50c07d082dd5be150718de858a4a5c565"
Sep 11 17:40:50.537782 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:40:50.537761    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b3b9f05a4b77a34c70195b36c715b50c07d082dd5be150718de858a4a5c565\": container with ID starting with d5b3b9f05a4b77a34c70195b36c715b50c07d082dd5be150718de858a4a5c565 not found: ID does not exist" containerID="d5b3b9f05a4b77a34c70195b36c715b50c07d082dd5be150718de858a4a5c565"
Sep 11 17:40:50.537839 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.537794    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b3b9f05a4b77a34c70195b36c715b50c07d082dd5be150718de858a4a5c565"} err="failed to get container status \"d5b3b9f05a4b77a34c70195b36c715b50c07d082dd5be150718de858a4a5c565\": rpc error: code = NotFound desc = could not find container \"d5b3b9f05a4b77a34c70195b36c715b50c07d082dd5be150718de858a4a5c565\": container with ID starting with d5b3b9f05a4b77a34c70195b36c715b50c07d082dd5be150718de858a4a5c565 not found: ID does not exist"
Sep 11 17:40:50.550566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.550516    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"]
Sep 11 17:40:50.554492 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.554445    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3d144889/e2e-lifecycle-model-3d144889-kserve-5fb5f95cff-hz94g"]
Sep 11 17:40:50.595959 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.595925    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:40:50.595959 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.595953    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:40:50.595959 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:50.595962    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hln9b\" (UniqueName: \"kubernetes.io/projected/07df38fd-c576-4711-a6bb-8bb0e9adbc9d-kube-api-access-hln9b\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:40:52.051452 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:40:52.051412    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07df38fd-c576-4711-a6bb-8bb0e9adbc9d" path="/var/lib/kubelet/pods/07df38fd-c576-4711-a6bb-8bb0e9adbc9d/volumes"
Sep 11 17:41:02.133885 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:02.133850    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:02.235157 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:02.235113    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:11.380342 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:11.380312    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:12.429751 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:12.429717    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:18.154594 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:18.154556    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:18.436548 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:18.432066    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"]
Sep 11 17:41:18.436548 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:18.432402    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl" podUID="2855f490-c549-4dd0-b96c-7f7c82a67118" containerName="istio-proxy" containerID="cri-o://4323db9f6fe96c746e6d2e8036e13aa0aca23cacb121e67e0424a50aaba867df" gracePeriod=30
Sep 11 17:41:19.466667 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:19.466624    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:19.730009 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:19.729912    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:19.829774 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:19.829714    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:19.929665 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:19.929634    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:20.566095 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:20.566056    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:20.658922 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:20.658891    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:20.759293 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:20.759253    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:22.025030 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:22.024996    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:23.690124 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.690097    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:41:23.709016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.708923    2560 generic.go:356] "Generic (PLEG): container finished" podID="2855f490-c549-4dd0-b96c-7f7c82a67118" containerID="4323db9f6fe96c746e6d2e8036e13aa0aca23cacb121e67e0424a50aaba867df" exitCode=0
Sep 11 17:41:23.709016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.708974    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"
Sep 11 17:41:23.709016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.708987    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl" event={"ID":"2855f490-c549-4dd0-b96c-7f7c82a67118","Type":"ContainerDied","Data":"4323db9f6fe96c746e6d2e8036e13aa0aca23cacb121e67e0424a50aaba867df"}
Sep 11 17:41:23.709237 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.709034    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl" event={"ID":"2855f490-c549-4dd0-b96c-7f7c82a67118","Type":"ContainerDied","Data":"76c6d683ff73157dc4b4e00009b12a8a9549e65ed682f233a61d4fda81ee5f75"}
Sep 11 17:41:23.709237 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.709058    2560 scope.go:117] "RemoveContainer" containerID="4323db9f6fe96c746e6d2e8036e13aa0aca23cacb121e67e0424a50aaba867df"
Sep 11 17:41:23.720908 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.720880    2560 scope.go:117] "RemoveContainer" containerID="7e675439449f2df5a80f3c21e7d081dc260f381d05519b0be1c5f6570ffa92e9"
Sep 11 17:41:23.730691 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.730666    2560 scope.go:117] "RemoveContainer" containerID="4323db9f6fe96c746e6d2e8036e13aa0aca23cacb121e67e0424a50aaba867df"
Sep 11 17:41:23.731082 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:41:23.731058    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4323db9f6fe96c746e6d2e8036e13aa0aca23cacb121e67e0424a50aaba867df\": container with ID starting with 4323db9f6fe96c746e6d2e8036e13aa0aca23cacb121e67e0424a50aaba867df not found: ID does not exist" containerID="4323db9f6fe96c746e6d2e8036e13aa0aca23cacb121e67e0424a50aaba867df"
Sep 11 17:41:23.731148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.731093    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4323db9f6fe96c746e6d2e8036e13aa0aca23cacb121e67e0424a50aaba867df"} err="failed to get container status \"4323db9f6fe96c746e6d2e8036e13aa0aca23cacb121e67e0424a50aaba867df\": rpc error: code = NotFound desc = could not find container \"4323db9f6fe96c746e6d2e8036e13aa0aca23cacb121e67e0424a50aaba867df\": container with ID starting with 4323db9f6fe96c746e6d2e8036e13aa0aca23cacb121e67e0424a50aaba867df not found: ID does not exist"
Sep 11 17:41:23.731148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.731113    2560 scope.go:117] "RemoveContainer" containerID="7e675439449f2df5a80f3c21e7d081dc260f381d05519b0be1c5f6570ffa92e9"
Sep 11 17:41:23.731452 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:41:23.731432    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e675439449f2df5a80f3c21e7d081dc260f381d05519b0be1c5f6570ffa92e9\": container with ID starting with 7e675439449f2df5a80f3c21e7d081dc260f381d05519b0be1c5f6570ffa92e9 not found: ID does not exist" containerID="7e675439449f2df5a80f3c21e7d081dc260f381d05519b0be1c5f6570ffa92e9"
Sep 11 17:41:23.731557 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.731454    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e675439449f2df5a80f3c21e7d081dc260f381d05519b0be1c5f6570ffa92e9"} err="failed to get container status \"7e675439449f2df5a80f3c21e7d081dc260f381d05519b0be1c5f6570ffa92e9\": rpc error: code = NotFound desc = could not find container \"7e675439449f2df5a80f3c21e7d081dc260f381d05519b0be1c5f6570ffa92e9\": container with ID starting with 7e675439449f2df5a80f3c21e7d081dc260f381d05519b0be1c5f6570ffa92e9 not found: ID does not exist"
Sep 11 17:41:23.760887 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.760852    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-envoy\") pod \"2855f490-c549-4dd0-b96c-7f7c82a67118\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") "
Sep 11 17:41:23.760887 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.760892    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-credential-socket\") pod \"2855f490-c549-4dd0-b96c-7f7c82a67118\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") "
Sep 11 17:41:23.761144 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.760911    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2855f490-c549-4dd0-b96c-7f7c82a67118-istiod-ca-cert\") pod \"2855f490-c549-4dd0-b96c-7f7c82a67118\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") "
Sep 11 17:41:23.761144 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.760971    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-token\") pod \"2855f490-c549-4dd0-b96c-7f7c82a67118\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") "
Sep 11 17:41:23.761144 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.761018    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs4wv\" (UniqueName: \"kubernetes.io/projected/2855f490-c549-4dd0-b96c-7f7c82a67118-kube-api-access-xs4wv\") pod \"2855f490-c549-4dd0-b96c-7f7c82a67118\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") "
Sep 11 17:41:23.761144 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.761048    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-data\") pod \"2855f490-c549-4dd0-b96c-7f7c82a67118\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") "
Sep 11 17:41:23.761144 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.761080    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-podinfo\") pod \"2855f490-c549-4dd0-b96c-7f7c82a67118\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") "
Sep 11 17:41:23.761144 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.761121    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-workload-socket\") pod \"2855f490-c549-4dd0-b96c-7f7c82a67118\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") "
Sep 11 17:41:23.761512 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.761185    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-workload-certs\") pod \"2855f490-c549-4dd0-b96c-7f7c82a67118\" (UID: \"2855f490-c549-4dd0-b96c-7f7c82a67118\") "
Sep 11 17:41:23.761512 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.761261    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2855f490-c549-4dd0-b96c-7f7c82a67118" (UID: "2855f490-c549-4dd0-b96c-7f7c82a67118"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:41:23.761630 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.761532    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2855f490-c549-4dd0-b96c-7f7c82a67118" (UID: "2855f490-c549-4dd0-b96c-7f7c82a67118"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:41:23.761630 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.761544    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2855f490-c549-4dd0-b96c-7f7c82a67118-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2855f490-c549-4dd0-b96c-7f7c82a67118" (UID: "2855f490-c549-4dd0-b96c-7f7c82a67118"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:41:23.761734 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.761653    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-workload-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:41:23.761734 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.761654    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-data" (OuterVolumeSpecName: "istio-data") pod "2855f490-c549-4dd0-b96c-7f7c82a67118" (UID: "2855f490-c549-4dd0-b96c-7f7c82a67118"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:41:23.761734 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.761672    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-credential-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:41:23.761734 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.761725    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2855f490-c549-4dd0-b96c-7f7c82a67118" (UID: "2855f490-c549-4dd0-b96c-7f7c82a67118"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:41:23.761910 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.761730    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2855f490-c549-4dd0-b96c-7f7c82a67118-istiod-ca-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:41:23.763848 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.763818    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2855f490-c549-4dd0-b96c-7f7c82a67118" (UID: "2855f490-c549-4dd0-b96c-7f7c82a67118"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:41:23.763989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.763847    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2855f490-c549-4dd0-b96c-7f7c82a67118" (UID: "2855f490-c549-4dd0-b96c-7f7c82a67118"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:41:23.763989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.763857    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-token" (OuterVolumeSpecName: "istio-token") pod "2855f490-c549-4dd0-b96c-7f7c82a67118" (UID: "2855f490-c549-4dd0-b96c-7f7c82a67118"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:41:23.763989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.763845    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2855f490-c549-4dd0-b96c-7f7c82a67118-kube-api-access-xs4wv" (OuterVolumeSpecName: "kube-api-access-xs4wv") pod "2855f490-c549-4dd0-b96c-7f7c82a67118" (UID: "2855f490-c549-4dd0-b96c-7f7c82a67118"). InnerVolumeSpecName "kube-api-access-xs4wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:41:23.862885 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.862838    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-envoy\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:41:23.862885 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.862874    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-token\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:41:23.862885 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.862885    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs4wv\" (UniqueName: \"kubernetes.io/projected/2855f490-c549-4dd0-b96c-7f7c82a67118-kube-api-access-xs4wv\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:41:23.862885 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.862893    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-data\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:41:23.863177 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.862903    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2855f490-c549-4dd0-b96c-7f7c82a67118-istio-podinfo\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:41:23.863177 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:23.862914    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2855f490-c549-4dd0-b96c-7f7c82a67118-workload-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:41:24.034876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:24.034842    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"]
Sep 11 17:41:24.042147 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:24.042099    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3d144889-openshift-default-67bbb5cb88-mbmfl"]
Sep 11 17:41:24.052279 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:24.052232    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2855f490-c549-4dd0-b96c-7f7c82a67118" path="/var/lib/kubelet/pods/2855f490-c549-4dd0-b96c-7f7c82a67118/volumes"
Sep 11 17:41:28.464163 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:28.464129    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:28.563394 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:28.563357    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:29.155965 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:29.155932    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:30.213312 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:30.213281    2560 scope.go:117] "RemoveContainer" containerID="bee0d5a1b54be99fe60624dd781d1979d060fcc5e71dd8051389cb163155a6bb"
Sep 11 17:41:30.221883 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:30.221859    2560 scope.go:117] "RemoveContainer" containerID="a26e38c5b5dece80e15f97ad8dabd7dd68e7248a377894eb23fe2a6cd1aa5818"
Sep 11 17:41:30.230593 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:30.230573    2560 scope.go:117] "RemoveContainer" containerID="0441dfbf7498babf1743966c598bbe3429161f7a2f0f3eb21952481c0bc377f6"
Sep 11 17:41:34.954624 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:34.954580    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:35.018713 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:35.018682    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:35.126859 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:35.126821    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:35.229307 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:35.229218    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:45.672036 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:45.671999    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:41:45.953113 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:41:45.953029    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:06.819990 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:06.819934    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:12.924088 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:12.924043    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:13.019656 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.019621    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:13.301773 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.301640    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:13.727774 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.727733    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"]
Sep 11 17:42:13.728165 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.728153    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2855f490-c549-4dd0-b96c-7f7c82a67118" containerName="istio-proxy"
Sep 11 17:42:13.728211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.728167    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2855f490-c549-4dd0-b96c-7f7c82a67118" containerName="istio-proxy"
Sep 11 17:42:13.728211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.728179    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07df38fd-c576-4711-a6bb-8bb0e9adbc9d" containerName="main"
Sep 11 17:42:13.728211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.728185    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="07df38fd-c576-4711-a6bb-8bb0e9adbc9d" containerName="main"
Sep 11 17:42:13.728211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.728199    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2855f490-c549-4dd0-b96c-7f7c82a67118" containerName="istio-proxy"
Sep 11 17:42:13.728211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.728205    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2855f490-c549-4dd0-b96c-7f7c82a67118" containerName="istio-proxy"
Sep 11 17:42:13.728368 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.728263    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2855f490-c549-4dd0-b96c-7f7c82a67118" containerName="istio-proxy"
Sep 11 17:42:13.728368 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.728271    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2855f490-c549-4dd0-b96c-7f7c82a67118" containerName="istio-proxy"
Sep 11 17:42:13.728368 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.728278    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="07df38fd-c576-4711-a6bb-8bb0e9adbc9d" containerName="main"
Sep 11 17:42:13.731467 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.731448    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.744594 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.744567    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-40f1fc-openshift-default-dockercfg-vlwvr\""
Sep 11 17:42:13.769244 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.769206    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"]
Sep 11 17:42:13.774499 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.771907    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istiod-ca-cert\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.774499 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.771957    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-envoy\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.774499 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.772000    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zv2m\" (UniqueName: \"kubernetes.io/projected/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-kube-api-access-8zv2m\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.774499 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.772025    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-podinfo\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.774499 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.772061    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-credential-socket\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.774499 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.772111    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-workload-socket\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.774499 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.772250    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-workload-certs\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.774499 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.772275    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-token\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.774499 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.772312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-data\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.790842 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.790805    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:13.873355 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.873315    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-credential-socket\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.873611 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.873367    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-workload-socket\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.873611 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.873445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-workload-certs\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.873611 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.873462    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-token\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.873611 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.873540    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-data\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.873611 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.873573    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istiod-ca-cert\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.873611 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.873591    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-envoy\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.873953 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.873627    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zv2m\" (UniqueName: \"kubernetes.io/projected/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-kube-api-access-8zv2m\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.873953 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.873653    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-podinfo\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.873953 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.873814    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-credential-socket\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.873953 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.873870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-workload-socket\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.873953 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.873915    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-workload-certs\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.874136 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.873965    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-data\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.874613 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.874589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istiod-ca-cert\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.875961 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.875940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-envoy\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.876069 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.876051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-podinfo\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.883103 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.883069    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-token\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:13.883807 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:13.883787    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zv2m\" (UniqueName: \"kubernetes.io/projected/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-kube-api-access-8zv2m\") pod \"e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") " pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:14.042010 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:14.041898    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:14.333908 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:14.333877    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"]
Sep 11 17:42:14.335699 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:42:14.335658    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50bf805_a6c5_4fc8_8dd4_2cbedee6a4be.slice/crio-c3cb15ea23b9ac0ea0b9aeec13370923b9a1fde0ad32c8a2e5241e97aa9ef1ac WatchSource:0}: Error finding container c3cb15ea23b9ac0ea0b9aeec13370923b9a1fde0ad32c8a2e5241e97aa9ef1ac: Status 404 returned error can't find the container with id c3cb15ea23b9ac0ea0b9aeec13370923b9a1fde0ad32c8a2e5241e97aa9ef1ac
Sep 11 17:42:14.336954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:14.336928    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:14.338096 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:14.338064    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:42:14.338175 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:14.338136    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:42:14.338175 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:14.338171    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:42:14.878491 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:14.878441    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7729-fa6034"]
Sep 11 17:42:14.882716 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:14.882676    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-fa6034"
Sep 11 17:42:14.888325 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:14.888291    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jtt66\""
Sep 11 17:42:14.890681 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:14.890654    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7729-fa6034"]
Sep 11 17:42:14.985865 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:14.985824    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mtqn\" (UniqueName: \"kubernetes.io/projected/d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a-kube-api-access-4mtqn\") pod \"test-curl-7729-fa6034\" (UID: \"d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a\") " pod="opendatahub/test-curl-7729-fa6034"
Sep 11 17:42:14.995097 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:14.995049    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s" event={"ID":"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be","Type":"ContainerStarted","Data":"d110fbb2a9fcd4a79a7289a11c00931e416b8e411fb0d1923e9f2462689d319f"}
Sep 11 17:42:14.995275 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:14.995118    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s" event={"ID":"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be","Type":"ContainerStarted","Data":"c3cb15ea23b9ac0ea0b9aeec13370923b9a1fde0ad32c8a2e5241e97aa9ef1ac"}
Sep 11 17:42:15.021774 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:15.021721    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s" podStartSLOduration=2.021700951 podStartE2EDuration="2.021700951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:42:15.017976318 +0000 UTC m=+1305.450429131" watchObservedRunningTime="2026-09-11 17:42:15.021700951 +0000 UTC m=+1305.454153725"
Sep 11 17:42:15.042413 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:15.042372    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:15.048346 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:15.048315    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:15.087235 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:15.087188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mtqn\" (UniqueName: \"kubernetes.io/projected/d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a-kube-api-access-4mtqn\") pod \"test-curl-7729-fa6034\" (UID: \"d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a\") " pod="opendatahub/test-curl-7729-fa6034"
Sep 11 17:42:15.097367 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:15.097333    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mtqn\" (UniqueName: \"kubernetes.io/projected/d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a-kube-api-access-4mtqn\") pod \"test-curl-7729-fa6034\" (UID: \"d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a\") " pod="opendatahub/test-curl-7729-fa6034"
Sep 11 17:42:15.197902 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:15.197863    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-fa6034"
Sep 11 17:42:15.481138 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:15.481113    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7729-fa6034"]
Sep 11 17:42:15.482808 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:42:15.482774    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd645dbb8_56fe_4fb5_a0e0_0bbde0dab29a.slice/crio-af9da351479fb84c6717432effabd503a07a80e58d18a4841d1e7fcf06815788 WatchSource:0}: Error finding container af9da351479fb84c6717432effabd503a07a80e58d18a4841d1e7fcf06815788: Status 404 returned error can't find the container with id af9da351479fb84c6717432effabd503a07a80e58d18a4841d1e7fcf06815788
Sep 11 17:42:16.003058 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:16.003016    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7729-fa6034" event={"ID":"d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a","Type":"ContainerStarted","Data":"e410dfac1670e42008faa98feb7356977de87ae80f746a10e3e19c3b8dccd281"}
Sep 11 17:42:16.003319 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:16.003072    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7729-fa6034" event={"ID":"d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a","Type":"ContainerStarted","Data":"af9da351479fb84c6717432effabd503a07a80e58d18a4841d1e7fcf06815788"}
Sep 11 17:42:16.003405 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:16.003381    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:16.004257 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:16.004234    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:16.021570 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:16.021501    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7729-fa6034" podStartSLOduration=2.021459501 podStartE2EDuration="2.021459501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:42:16.01982018 +0000 UTC m=+1306.452272958" watchObservedRunningTime="2026-09-11 17:42:16.021459501 +0000 UTC m=+1306.453912278"
Sep 11 17:42:16.251105 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:16.251065    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:16.707234 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:16.707143    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7729-fa6034"]
Sep 11 17:42:16.713441 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:16.713410    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7729-fa6034"]
Sep 11 17:42:16.949165 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:16.949127    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:17.009325 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:42:17.009230    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7729-fa6034" container="test-curl-7729-fa6034"
Sep 11 17:42:17.057521 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:17.057442    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:17.173682 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:17.173614    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:17.947279 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:17.947244    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:18.016082 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:18.016044    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7729-fa6034" podUID="d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a" containerName="test-curl-7729-fa6034" containerID="cri-o://e410dfac1670e42008faa98feb7356977de87ae80f746a10e3e19c3b8dccd281" gracePeriod=2
Sep 11 17:42:18.019256 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:42:18.019221    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7729-fa6034\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a" pod="opendatahub/test-curl-7729-fa6034"
Sep 11 17:42:18.048846 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:18.048810    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:18.228153 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:18.228065    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"]
Sep 11 17:42:18.247089 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:18.247045    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:18.346181 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:18.346147    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:19.112517 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:19.112455    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:19.293404 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:19.293370    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"]
Sep 11 17:42:19.311756 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:19.311724    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:20.048873 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:20.048835    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:20.202611 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:20.202582    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-fa6034"
Sep 11 17:42:20.205068 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:42:20.205037    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7729-fa6034\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a" pod="opendatahub/test-curl-7729-fa6034"
Sep 11 17:42:20.242567 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:20.242531    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mtqn\" (UniqueName: \"kubernetes.io/projected/d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a-kube-api-access-4mtqn\") pod \"d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a\" (UID: \"d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a\") "
Sep 11 17:42:20.244625 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:20.244596    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a-kube-api-access-4mtqn" (OuterVolumeSpecName: "kube-api-access-4mtqn") pod "d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a" (UID: "d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a"). InnerVolumeSpecName "kube-api-access-4mtqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:42:20.343516 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:20.343397    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mtqn\" (UniqueName: \"kubernetes.io/projected/d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a-kube-api-access-4mtqn\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:42:20.849719 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:20.849682    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:20.950931 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:20.950893    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:21.035285 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:21.035251    2560 generic.go:356] "Generic (PLEG): container finished" podID="d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a" containerID="e410dfac1670e42008faa98feb7356977de87ae80f746a10e3e19c3b8dccd281" exitCode=137
Sep 11 17:42:21.035469 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:21.035298    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-fa6034"
Sep 11 17:42:21.035469 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:21.035315    2560 scope.go:117] "RemoveContainer" containerID="e410dfac1670e42008faa98feb7356977de87ae80f746a10e3e19c3b8dccd281"
Sep 11 17:42:21.038404 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:42:21.038374    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7729-fa6034\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a" pod="opendatahub/test-curl-7729-fa6034"
Sep 11 17:42:21.044955 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:21.044937    2560 scope.go:117] "RemoveContainer" containerID="e410dfac1670e42008faa98feb7356977de87ae80f746a10e3e19c3b8dccd281"
Sep 11 17:42:21.045310 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:42:21.045284    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e410dfac1670e42008faa98feb7356977de87ae80f746a10e3e19c3b8dccd281\": container with ID starting with e410dfac1670e42008faa98feb7356977de87ae80f746a10e3e19c3b8dccd281 not found: ID does not exist" containerID="e410dfac1670e42008faa98feb7356977de87ae80f746a10e3e19c3b8dccd281"
Sep 11 17:42:21.045383 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:21.045318    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e410dfac1670e42008faa98feb7356977de87ae80f746a10e3e19c3b8dccd281"} err="failed to get container status \"e410dfac1670e42008faa98feb7356977de87ae80f746a10e3e19c3b8dccd281\": rpc error: code = NotFound desc = could not find container \"e410dfac1670e42008faa98feb7356977de87ae80f746a10e3e19c3b8dccd281\": container with ID starting with e410dfac1670e42008faa98feb7356977de87ae80f746a10e3e19c3b8dccd281 not found: ID does not exist"
Sep 11 17:42:21.045886 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:42:21.045862    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7729-fa6034\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a" pod="opendatahub/test-curl-7729-fa6034"
Sep 11 17:42:21.220395 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:21.220361    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:21.343429 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:21.343398    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:21.435981 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:21.435946    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:22.051383 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:22.051351    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a" path="/var/lib/kubelet/pods/d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a/volumes"
Sep 11 17:42:26.423691 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.423648    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"]
Sep 11 17:42:26.424233 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.424212    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a" containerName="test-curl-7729-fa6034"
Sep 11 17:42:26.424326 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.424236    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a" containerName="test-curl-7729-fa6034"
Sep 11 17:42:26.424383 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.424368    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d645dbb8-56fe-4fb5-a0e0-0bbde0dab29a" containerName="test-curl-7729-fa6034"
Sep 11 17:42:26.428868 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.428847    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.431595 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.431573    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4c36f4\"/\"sub-test-model-4c36f4-kserve-self-signed-certs\""
Sep 11 17:42:26.441567 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.441545    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"]
Sep 11 17:42:26.450405 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.450370    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:26.502942 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.502907    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-dshm\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.502942 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.502939    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-home\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.503166 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.502969    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-model-cache\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.503166 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.503053    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/434404c1-2a0e-4c82-ad92-05f53eb21111-tls-certs\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.503166 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.503127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-tmp-dir\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.503285 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.503202    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z858b\" (UniqueName: \"kubernetes.io/projected/434404c1-2a0e-4c82-ad92-05f53eb21111-kube-api-access-z858b\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.605155 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.605115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-tmp-dir\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.605332 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.604391    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-tmp-dir\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.605332 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.605311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z858b\" (UniqueName: \"kubernetes.io/projected/434404c1-2a0e-4c82-ad92-05f53eb21111-kube-api-access-z858b\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.605555 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.605367    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-dshm\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.605555 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.605394    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-home\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.605555 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.605439    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-model-cache\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.605555 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.605500    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/434404c1-2a0e-4c82-ad92-05f53eb21111-tls-certs\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.606411 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.606370    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-home\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.606561 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.606464    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-model-cache\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.610984 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.610306    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-dshm\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.611187 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.610996    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/434404c1-2a0e-4c82-ad92-05f53eb21111-tls-certs\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.617177 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.617154    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z858b\" (UniqueName: \"kubernetes.io/projected/434404c1-2a0e-4c82-ad92-05f53eb21111-kube-api-access-z858b\") pod \"sub-test-model-4c36f4-kserve-8474569579-qgh4v\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:26.739502 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:26.739370    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:27.017486 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:27.017452    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"]
Sep 11 17:42:27.019320 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:42:27.019292    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod434404c1_2a0e_4c82_ad92_05f53eb21111.slice/crio-72e4182bc9933cf851294b1ac7ac06576dc46244d1bb579b075699b61c33d40e WatchSource:0}: Error finding container 72e4182bc9933cf851294b1ac7ac06576dc46244d1bb579b075699b61c33d40e: Status 404 returned error can't find the container with id 72e4182bc9933cf851294b1ac7ac06576dc46244d1bb579b075699b61c33d40e
Sep 11 17:42:27.068886 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:27.068852    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v" event={"ID":"434404c1-2a0e-4c82-ad92-05f53eb21111","Type":"ContainerStarted","Data":"72e4182bc9933cf851294b1ac7ac06576dc46244d1bb579b075699b61c33d40e"}
Sep 11 17:42:28.076204 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:28.076156    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v" event={"ID":"434404c1-2a0e-4c82-ad92-05f53eb21111","Type":"ContainerStarted","Data":"fd9aaa9c9b06e90a6748be595713289ce83567513facb1bc0f3b88fd3ae94ad3"}
Sep 11 17:42:28.095101 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:28.095047    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v" podStartSLOduration=2.095028251 podStartE2EDuration="2.095028251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:42:28.093205214 +0000 UTC m=+1318.525657992" watchObservedRunningTime="2026-09-11 17:42:28.095028251 +0000 UTC m=+1318.527481065"
Sep 11 17:42:33.249246 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.249200    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb" podUID="54cbbc5f-92a1-447d-b739-4ccbfccf6d43" containerName="main" containerID="cri-o://a1778a6d59cd586cf3bd74858380c0a869f02ec9bd7b2641db6a33d325a8fc59" gracePeriod=45
Sep 11 17:42:33.543973 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.543946    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:42:33.688092 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.688057    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7h64g\" (UniqueName: \"kubernetes.io/projected/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-kube-api-access-7h64g\") pod \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") "
Sep 11 17:42:33.688092 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.688106    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-home\") pod \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") "
Sep 11 17:42:33.688372 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.688155    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-tmp-dir\") pod \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") "
Sep 11 17:42:33.688372 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.688178    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-dshm\") pod \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") "
Sep 11 17:42:33.688372 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.688201    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-model-cache\") pod \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") "
Sep 11 17:42:33.688372 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.688275    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-tls-certs\") pod \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\" (UID: \"54cbbc5f-92a1-447d-b739-4ccbfccf6d43\") "
Sep 11 17:42:33.688636 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.688429    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54cbbc5f-92a1-447d-b739-4ccbfccf6d43" (UID: "54cbbc5f-92a1-447d-b739-4ccbfccf6d43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:42:33.688636 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.688468    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-home" (OuterVolumeSpecName: "home") pod "54cbbc5f-92a1-447d-b739-4ccbfccf6d43" (UID: "54cbbc5f-92a1-447d-b739-4ccbfccf6d43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:42:33.688636 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.688545    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-model-cache" (OuterVolumeSpecName: "model-cache") pod "54cbbc5f-92a1-447d-b739-4ccbfccf6d43" (UID: "54cbbc5f-92a1-447d-b739-4ccbfccf6d43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:42:33.688753 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.688714    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:42:33.688753 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.688728    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:42:33.688753 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.688738    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:42:33.690616 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.690575    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54cbbc5f-92a1-447d-b739-4ccbfccf6d43" (UID: "54cbbc5f-92a1-447d-b739-4ccbfccf6d43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:42:33.690879 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.690861    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-dshm" (OuterVolumeSpecName: "dshm") pod "54cbbc5f-92a1-447d-b739-4ccbfccf6d43" (UID: "54cbbc5f-92a1-447d-b739-4ccbfccf6d43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:42:33.690948 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.690923    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-kube-api-access-7h64g" (OuterVolumeSpecName: "kube-api-access-7h64g") pod "54cbbc5f-92a1-447d-b739-4ccbfccf6d43" (UID: "54cbbc5f-92a1-447d-b739-4ccbfccf6d43"). InnerVolumeSpecName "kube-api-access-7h64g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:42:33.789469 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.789367    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h64g\" (UniqueName: \"kubernetes.io/projected/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-kube-api-access-7h64g\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:42:33.789469 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.789406    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:42:33.789469 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:33.789415    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54cbbc5f-92a1-447d-b739-4ccbfccf6d43-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:42:34.119909 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.118686    2560 generic.go:356] "Generic (PLEG): container finished" podID="54cbbc5f-92a1-447d-b739-4ccbfccf6d43" containerID="a1778a6d59cd586cf3bd74858380c0a869f02ec9bd7b2641db6a33d325a8fc59" exitCode=0
Sep 11 17:42:34.119909 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.118757    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb" event={"ID":"54cbbc5f-92a1-447d-b739-4ccbfccf6d43","Type":"ContainerDied","Data":"a1778a6d59cd586cf3bd74858380c0a869f02ec9bd7b2641db6a33d325a8fc59"}
Sep 11 17:42:34.119909 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.118793    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb" event={"ID":"54cbbc5f-92a1-447d-b739-4ccbfccf6d43","Type":"ContainerDied","Data":"848d92abf5a31245145794e02d6f3bc9a99e2cad02ac997fdffcdbdabb481757"}
Sep 11 17:42:34.119909 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.118820    2560 scope.go:117] "RemoveContainer" containerID="a1778a6d59cd586cf3bd74858380c0a869f02ec9bd7b2641db6a33d325a8fc59"
Sep 11 17:42:34.119909 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.118988    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"
Sep 11 17:42:34.136816 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.136790    2560 scope.go:117] "RemoveContainer" containerID="a1778a6d59cd586cf3bd74858380c0a869f02ec9bd7b2641db6a33d325a8fc59"
Sep 11 17:42:34.137218 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:42:34.137188    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1778a6d59cd586cf3bd74858380c0a869f02ec9bd7b2641db6a33d325a8fc59\": container with ID starting with a1778a6d59cd586cf3bd74858380c0a869f02ec9bd7b2641db6a33d325a8fc59 not found: ID does not exist" containerID="a1778a6d59cd586cf3bd74858380c0a869f02ec9bd7b2641db6a33d325a8fc59"
Sep 11 17:42:34.137320 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.137228    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1778a6d59cd586cf3bd74858380c0a869f02ec9bd7b2641db6a33d325a8fc59"} err="failed to get container status \"a1778a6d59cd586cf3bd74858380c0a869f02ec9bd7b2641db6a33d325a8fc59\": rpc error: code = NotFound desc = could not find container \"a1778a6d59cd586cf3bd74858380c0a869f02ec9bd7b2641db6a33d325a8fc59\": container with ID starting with a1778a6d59cd586cf3bd74858380c0a869f02ec9bd7b2641db6a33d325a8fc59 not found: ID does not exist"
Sep 11 17:42:34.138678 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.138656    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"]
Sep 11 17:42:34.142039 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.142014    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/auth-test-model-4c36f4-kserve-8468dff475-58dwb"]
Sep 11 17:42:34.316331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.316215    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j" podUID="68160d51-663c-4b0e-80d1-c9c924712f12" containerName="main" containerID="cri-o://a97987898ea58088f9246c40c4f262fad14802ad4c03b1bd15d7f681d30ba4af" gracePeriod=45
Sep 11 17:42:34.589401 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.589370    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:42:34.701092 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.701053    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-tmp-dir\") pod \"68160d51-663c-4b0e-80d1-c9c924712f12\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") "
Sep 11 17:42:34.701271 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.701125    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-model-cache\") pod \"68160d51-663c-4b0e-80d1-c9c924712f12\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") "
Sep 11 17:42:34.701271 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.701189    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxtcc\" (UniqueName: \"kubernetes.io/projected/68160d51-663c-4b0e-80d1-c9c924712f12-kube-api-access-jxtcc\") pod \"68160d51-663c-4b0e-80d1-c9c924712f12\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") "
Sep 11 17:42:34.701271 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.701209    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68160d51-663c-4b0e-80d1-c9c924712f12-tls-certs\") pod \"68160d51-663c-4b0e-80d1-c9c924712f12\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") "
Sep 11 17:42:34.701271 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.701228    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-home\") pod \"68160d51-663c-4b0e-80d1-c9c924712f12\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") "
Sep 11 17:42:34.701512 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.701274    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-dshm\") pod \"68160d51-663c-4b0e-80d1-c9c924712f12\" (UID: \"68160d51-663c-4b0e-80d1-c9c924712f12\") "
Sep 11 17:42:34.701512 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.701346    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68160d51-663c-4b0e-80d1-c9c924712f12" (UID: "68160d51-663c-4b0e-80d1-c9c924712f12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:42:34.701512 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.701386    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-model-cache" (OuterVolumeSpecName: "model-cache") pod "68160d51-663c-4b0e-80d1-c9c924712f12" (UID: "68160d51-663c-4b0e-80d1-c9c924712f12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:42:34.701512 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.701506    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-home" (OuterVolumeSpecName: "home") pod "68160d51-663c-4b0e-80d1-c9c924712f12" (UID: "68160d51-663c-4b0e-80d1-c9c924712f12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:42:34.701750 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.701648    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:42:34.701750 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.701659    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:42:34.701750 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.701667    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:42:34.703458 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.703436    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-dshm" (OuterVolumeSpecName: "dshm") pod "68160d51-663c-4b0e-80d1-c9c924712f12" (UID: "68160d51-663c-4b0e-80d1-c9c924712f12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:42:34.703566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.703533    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68160d51-663c-4b0e-80d1-c9c924712f12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68160d51-663c-4b0e-80d1-c9c924712f12" (UID: "68160d51-663c-4b0e-80d1-c9c924712f12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:42:34.703613 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.703600    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68160d51-663c-4b0e-80d1-c9c924712f12-kube-api-access-jxtcc" (OuterVolumeSpecName: "kube-api-access-jxtcc") pod "68160d51-663c-4b0e-80d1-c9c924712f12" (UID: "68160d51-663c-4b0e-80d1-c9c924712f12"). InnerVolumeSpecName "kube-api-access-jxtcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:42:34.802422 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.802388    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxtcc\" (UniqueName: \"kubernetes.io/projected/68160d51-663c-4b0e-80d1-c9c924712f12-kube-api-access-jxtcc\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:42:34.802422 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.802419    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68160d51-663c-4b0e-80d1-c9c924712f12-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:42:34.802422 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:34.802429    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68160d51-663c-4b0e-80d1-c9c924712f12-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:42:35.129359 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:35.129314    2560 generic.go:356] "Generic (PLEG): container finished" podID="68160d51-663c-4b0e-80d1-c9c924712f12" containerID="a97987898ea58088f9246c40c4f262fad14802ad4c03b1bd15d7f681d30ba4af" exitCode=0
Sep 11 17:42:35.129587 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:35.129492    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j" event={"ID":"68160d51-663c-4b0e-80d1-c9c924712f12","Type":"ContainerDied","Data":"a97987898ea58088f9246c40c4f262fad14802ad4c03b1bd15d7f681d30ba4af"}
Sep 11 17:42:35.129587 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:35.129549    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j" event={"ID":"68160d51-663c-4b0e-80d1-c9c924712f12","Type":"ContainerDied","Data":"b0e519b84fd346ad20b1f832eb1dd3e0b93f0afc8366d227058f1ffe2daca16e"}
Sep 11 17:42:35.129587 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:35.129571    2560 scope.go:117] "RemoveContainer" containerID="a97987898ea58088f9246c40c4f262fad14802ad4c03b1bd15d7f681d30ba4af"
Sep 11 17:42:35.129790 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:35.129621    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"
Sep 11 17:42:35.140367 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:35.140347    2560 scope.go:117] "RemoveContainer" containerID="a97987898ea58088f9246c40c4f262fad14802ad4c03b1bd15d7f681d30ba4af"
Sep 11 17:42:35.140721 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:42:35.140697    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a97987898ea58088f9246c40c4f262fad14802ad4c03b1bd15d7f681d30ba4af\": container with ID starting with a97987898ea58088f9246c40c4f262fad14802ad4c03b1bd15d7f681d30ba4af not found: ID does not exist" containerID="a97987898ea58088f9246c40c4f262fad14802ad4c03b1bd15d7f681d30ba4af"
Sep 11 17:42:35.140800 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:35.140733    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a97987898ea58088f9246c40c4f262fad14802ad4c03b1bd15d7f681d30ba4af"} err="failed to get container status \"a97987898ea58088f9246c40c4f262fad14802ad4c03b1bd15d7f681d30ba4af\": rpc error: code = NotFound desc = could not find container \"a97987898ea58088f9246c40c4f262fad14802ad4c03b1bd15d7f681d30ba4af\": container with ID starting with a97987898ea58088f9246c40c4f262fad14802ad4c03b1bd15d7f681d30ba4af not found: ID does not exist"
Sep 11 17:42:35.153351 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:35.153312    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"]
Sep 11 17:42:35.157275 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:35.157242    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/auth-test-model-db95fd-kserve-797646cd75-t4v2j"]
Sep 11 17:42:36.053301 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:36.053253    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54cbbc5f-92a1-447d-b739-4ccbfccf6d43" path="/var/lib/kubelet/pods/54cbbc5f-92a1-447d-b739-4ccbfccf6d43/volumes"
Sep 11 17:42:36.053984 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:36.053960    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68160d51-663c-4b0e-80d1-c9c924712f12" path="/var/lib/kubelet/pods/68160d51-663c-4b0e-80d1-c9c924712f12/volumes"
Sep 11 17:42:36.739536 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:36.739504    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:36.739709 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:36.739550    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:36.752484 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:36.752440    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:37.155272 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:37.155242    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:42:41.166933 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:41.166898    2560 generic.go:356] "Generic (PLEG): container finished" podID="b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" containerID="d110fbb2a9fcd4a79a7289a11c00931e416b8e411fb0d1923e9f2462689d319f" exitCode=0
Sep 11 17:42:41.167377 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:41.166971    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s" event={"ID":"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be","Type":"ContainerDied","Data":"d110fbb2a9fcd4a79a7289a11c00931e416b8e411fb0d1923e9f2462689d319f"}
Sep 11 17:42:41.167425 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:41.167396    2560 scope.go:117] "RemoveContainer" containerID="d110fbb2a9fcd4a79a7289a11c00931e416b8e411fb0d1923e9f2462689d319f"
Sep 11 17:42:41.168645 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:41.168612    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:42:41.168761 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:41.168697    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:42:41.168825 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:41.168758    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:42:42.181052 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:42.181011    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s" event={"ID":"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be","Type":"ContainerStarted","Data":"ddba8ea7ff13b36962ebdca7f78d8ef30eb75cfe87b472255f1152e644e37108"}
Sep 11 17:42:42.877084 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:42.877046    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:43.042466 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:43.042432    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:43.043568 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:43.043544    2560 patch_prober.go:28] interesting pod/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 11 17:42:43.043655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:43.043595    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s" podUID="b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 11 17:42:43.808553 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:43.808513    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:44.042969 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:44.042927    2560 patch_prober.go:28] interesting pod/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 11 17:42:44.043145 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:44.042984    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s" podUID="b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 11 17:42:44.534243 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:44.534210    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:45.043206 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:45.043164    2560 patch_prober.go:28] interesting pod/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 11 17:42:45.043654 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:45.043229    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s" podUID="b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 11 17:42:45.260049 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:45.260019    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:46.003183 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:46.003141    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:46.043220 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:46.043182    2560 patch_prober.go:28] interesting pod/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 11 17:42:46.043671 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:46.043252    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s" podUID="b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 11 17:42:46.052679 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:46.052651    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:47.024580 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.024463    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:47.042445 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.042398    2560 patch_prober.go:28] interesting pod/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 11 17:42:47.042672 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.042505    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s" podUID="b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 11 17:42:47.124529 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.124492    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:47.566420 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.566374    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:47.781075 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.781042    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"]
Sep 11 17:42:47.781445 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.781432    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54cbbc5f-92a1-447d-b739-4ccbfccf6d43" containerName="main"
Sep 11 17:42:47.781514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.781446    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="54cbbc5f-92a1-447d-b739-4ccbfccf6d43" containerName="main"
Sep 11 17:42:47.781514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.781469    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68160d51-663c-4b0e-80d1-c9c924712f12" containerName="main"
Sep 11 17:42:47.781514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.781506    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="68160d51-663c-4b0e-80d1-c9c924712f12" containerName="main"
Sep 11 17:42:47.781621 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.781585    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="54cbbc5f-92a1-447d-b739-4ccbfccf6d43" containerName="main"
Sep 11 17:42:47.781621 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.781592    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="68160d51-663c-4b0e-80d1-c9c924712f12" containerName="main"
Sep 11 17:42:47.783918 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.783900    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.787202 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.787180    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-86db1c-openshift-default-dockercfg-f4x59\""
Sep 11 17:42:47.797931 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.797903    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"]
Sep 11 17:42:47.827373 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.827283    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:47.855575 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.855541    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-workload-socket\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.855769 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.855604    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-token\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.855769 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.855653    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7102f469-a6f9-4502-82d4-dcab6acbfda3-istiod-ca-cert\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.855769 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.855716    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-workload-certs\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.855769 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.855754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwnwg\" (UniqueName: \"kubernetes.io/projected/7102f469-a6f9-4502-82d4-dcab6acbfda3-kube-api-access-wwnwg\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.855926 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.855777    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-podinfo\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.855926 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.855807    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-credential-socket\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.855926 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.855883    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-data\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.856025 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.855948    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-envoy\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.928400 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.928360    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:47.956959 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.956927    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-envoy\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.957109 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.956972    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-workload-socket\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.957109 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.957012    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-token\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.957109 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.957046    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7102f469-a6f9-4502-82d4-dcab6acbfda3-istiod-ca-cert\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.957109 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.957071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-workload-certs\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.957331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.957112    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwnwg\" (UniqueName: \"kubernetes.io/projected/7102f469-a6f9-4502-82d4-dcab6acbfda3-kube-api-access-wwnwg\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.957331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.957138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-podinfo\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.957331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.957179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-credential-socket\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.957331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.957207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-data\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.957581 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.957536    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-workload-socket\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.957857 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.957828    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-data\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.957977 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.957879    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-workload-certs\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.957977 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.957911    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-credential-socket\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.957977 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.957933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7102f469-a6f9-4502-82d4-dcab6acbfda3-istiod-ca-cert\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.959317 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.959296    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-envoy\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.959704 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.959679    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-podinfo\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.969118 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.969062    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-token\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:47.969670 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:47.969625    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwnwg\" (UniqueName: \"kubernetes.io/projected/7102f469-a6f9-4502-82d4-dcab6acbfda3-kube-api-access-wwnwg\") pod \"e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") " pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:48.051143 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:48.051111    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:48.052168 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:48.052146    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:42:48.094768 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:48.094664    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:48.399229 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:48.399205    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"]
Sep 11 17:42:48.401301 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:42:48.401272    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7102f469_a6f9_4502_82d4_dcab6acbfda3.slice/crio-29dee3d25ac540c49e6838adcf0c922e9fa5156e88d711992fbcbfbe0ed2f662 WatchSource:0}: Error finding container 29dee3d25ac540c49e6838adcf0c922e9fa5156e88d711992fbcbfbe0ed2f662: Status 404 returned error can't find the container with id 29dee3d25ac540c49e6838adcf0c922e9fa5156e88d711992fbcbfbe0ed2f662
Sep 11 17:42:48.403502 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:48.403454    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:42:48.403602 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:48.403552    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:42:48.403674 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:48.403597    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:42:49.232832 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:49.232798    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64" event={"ID":"7102f469-a6f9-4502-82d4-dcab6acbfda3","Type":"ContainerStarted","Data":"53bf18523fa0b96e18358e65239c32e8b0bdb99e05d4c3ef2541d13bb1e86f33"}
Sep 11 17:42:49.233038 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:49.232854    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64" event={"ID":"7102f469-a6f9-4502-82d4-dcab6acbfda3","Type":"ContainerStarted","Data":"29dee3d25ac540c49e6838adcf0c922e9fa5156e88d711992fbcbfbe0ed2f662"}
Sep 11 17:42:49.269317 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:49.269259    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64" podStartSLOduration=2.269241909 podStartE2EDuration="2.269241909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:42:49.267657616 +0000 UTC m=+1339.700110397" watchObservedRunningTime="2026-09-11 17:42:49.269241909 +0000 UTC m=+1339.701694953"
Sep 11 17:42:50.095202 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:50.095170    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:50.100173 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:50.100151    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:50.238102 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:50.238073    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:50.239062 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:50.239031    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:42:50.850247 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:50.850187    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:51.371940 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:51.371891    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:51.482907 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:51.482877    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:52.146611 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:52.146460    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:53.155406 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:53.155368    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:54.177631 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:54.177582    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:55.176375 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:55.176339    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:55.937722 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:55.937682    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:56.883358 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:56.883326    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:56.996229 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:56.996196    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:57.231673 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:57.231583    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:57.544798 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:57.544688    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:42:57.642965 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:42:57.642929    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:02.713525 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:02.713488    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:08.136868 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.136824    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"]
Sep 11 17:43:08.141175 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.141141    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.145389 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.145357    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-db95fd\"/\"sub-test-model-db95fd-kserve-self-signed-certs\""
Sep 11 17:43:08.145711 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.145693    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-db95fd\"/\"default-dockercfg-fhfrq\""
Sep 11 17:43:08.145923 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.145909    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-db95fd\"/\"openshift-service-ca.crt\""
Sep 11 17:43:08.146159 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.146139    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-db95fd\"/\"kube-root-ca.crt\""
Sep 11 17:43:08.148347 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.148320    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"]
Sep 11 17:43:08.153733 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.153706    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:08.277971 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.277926    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q74d7\" (UniqueName: \"kubernetes.io/projected/7ee86935-b26d-4e03-8af2-ba3c8ac78055-kube-api-access-q74d7\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.278170 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.277991    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-model-cache\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.278170 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.278059    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-tmp-dir\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.278170 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.278103    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee86935-b26d-4e03-8af2-ba3c8ac78055-tls-certs\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.278170 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.278134    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-dshm\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.278339 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.278230    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-home\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.379698 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.379651    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee86935-b26d-4e03-8af2-ba3c8ac78055-tls-certs\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.379917 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.379724    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-dshm\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.379917 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.379786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-home\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.379917 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.379856    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q74d7\" (UniqueName: \"kubernetes.io/projected/7ee86935-b26d-4e03-8af2-ba3c8ac78055-kube-api-access-q74d7\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.379917 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.379914    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-model-cache\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.380140 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.379952    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-tmp-dir\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.380306 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.380279    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-home\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.380393 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.380314    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-model-cache\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.380393 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.380356    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-tmp-dir\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.382117 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.382097    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-dshm\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.382410 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.382390    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee86935-b26d-4e03-8af2-ba3c8ac78055-tls-certs\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.389185 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.389124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q74d7\" (UniqueName: \"kubernetes.io/projected/7ee86935-b26d-4e03-8af2-ba3c8ac78055-kube-api-access-q74d7\") pod \"sub-test-model-db95fd-kserve-849776bf6c-cv2cw\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.454874 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.454831    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:08.760249 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:08.760217    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"]
Sep 11 17:43:08.762229 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:43:08.762201    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ee86935_b26d_4e03_8af2_ba3c8ac78055.slice/crio-9749565a74bba427329b511dceb89026ef831f4c5fb45b65a246671571cea400 WatchSource:0}: Error finding container 9749565a74bba427329b511dceb89026ef831f4c5fb45b65a246671571cea400: Status 404 returned error can't find the container with id 9749565a74bba427329b511dceb89026ef831f4c5fb45b65a246671571cea400
Sep 11 17:43:09.359374 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:09.359335    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw" event={"ID":"7ee86935-b26d-4e03-8af2-ba3c8ac78055","Type":"ContainerStarted","Data":"2aa17915778cd6c0caf6fba3d56d71d521725da6a70d39cbcf820878b9f8c966"}
Sep 11 17:43:09.359374 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:09.359381    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw" event={"ID":"7ee86935-b26d-4e03-8af2-ba3c8ac78055","Type":"ContainerStarted","Data":"9749565a74bba427329b511dceb89026ef831f4c5fb45b65a246671571cea400"}
Sep 11 17:43:09.376989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:09.376931    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw" podStartSLOduration=1.376910531 podStartE2EDuration="1.376910531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:43:09.375917929 +0000 UTC m=+1359.808370707" watchObservedRunningTime="2026-09-11 17:43:09.376910531 +0000 UTC m=+1359.809363308"
Sep 11 17:43:13.383300 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:13.383257    2560 generic.go:356] "Generic (PLEG): container finished" podID="7102f469-a6f9-4502-82d4-dcab6acbfda3" containerID="53bf18523fa0b96e18358e65239c32e8b0bdb99e05d4c3ef2541d13bb1e86f33" exitCode=0
Sep 11 17:43:13.383845 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:13.383327    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64" event={"ID":"7102f469-a6f9-4502-82d4-dcab6acbfda3","Type":"ContainerDied","Data":"53bf18523fa0b96e18358e65239c32e8b0bdb99e05d4c3ef2541d13bb1e86f33"}
Sep 11 17:43:13.383845 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:13.383791    2560 scope.go:117] "RemoveContainer" containerID="53bf18523fa0b96e18358e65239c32e8b0bdb99e05d4c3ef2541d13bb1e86f33"
Sep 11 17:43:13.385076 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:13.385039    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:43:13.385174 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:13.385113    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:43:13.385174 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:13.385145    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:43:14.390639 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:14.390607    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64" event={"ID":"7102f469-a6f9-4502-82d4-dcab6acbfda3","Type":"ContainerStarted","Data":"30330c81adb656c23007542c8197e9b2023e97862eacfd684d7f0ed3ee712f55"}
Sep 11 17:43:15.095342 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:15.095306    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:43:15.095636 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:15.095610    2560 patch_prober.go:28] interesting pod/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 11 17:43:15.095687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:15.095662    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64" podUID="7102f469-a6f9-4502-82d4-dcab6acbfda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 11 17:43:16.095463 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:16.095415    2560 patch_prober.go:28] interesting pod/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 11 17:43:16.096027 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:16.095530    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64" podUID="7102f469-a6f9-4502-82d4-dcab6acbfda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 11 17:43:17.095567 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:17.095530    2560 patch_prober.go:28] interesting pod/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 11 17:43:17.095969 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:17.095589    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64" podUID="7102f469-a6f9-4502-82d4-dcab6acbfda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 11 17:43:18.095927 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:18.095887    2560 patch_prober.go:28] interesting pod/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 11 17:43:18.096330 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:18.095963    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64" podUID="7102f469-a6f9-4502-82d4-dcab6acbfda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 11 17:43:18.455506 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:18.455464    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:18.455687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:18.455520    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:18.468188 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:18.468160    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:19.095164 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:19.095122    2560 patch_prober.go:28] interesting pod/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 11 17:43:19.095353 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:19.095201    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64" podUID="7102f469-a6f9-4502-82d4-dcab6acbfda3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 11 17:43:19.441608 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:19.441580    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:43:19.911221 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:19.911192    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:20.098969 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:20.098938    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:43:20.099235 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:20.099214    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:43:20.100028 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:20.100011    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:43:21.283774 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:21.283737    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:21.600410 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:21.600326    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:22.845636 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:22.845602    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:24.275597 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:24.275557    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:24.679238 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:24.679194    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6"]
Sep 11 17:43:24.683877 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:24.683852    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6"
Sep 11 17:43:24.688110 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:24.688083    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:43:24.688110 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:24.688083    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:43:24.711236 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:24.711200    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6"]
Sep 11 17:43:24.760104 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:24.760070    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ef809a-4b68-4d84-8f0c-5585ee816aa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6\" (UID: \"33ef809a-4b68-4d84-8f0c-5585ee816aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6"
Sep 11 17:43:24.861182 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:24.861133    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ef809a-4b68-4d84-8f0c-5585ee816aa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6\" (UID: \"33ef809a-4b68-4d84-8f0c-5585ee816aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6"
Sep 11 17:43:24.861817 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:24.861795    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ef809a-4b68-4d84-8f0c-5585ee816aa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6\" (UID: \"33ef809a-4b68-4d84-8f0c-5585ee816aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6"
Sep 11 17:43:24.995415 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:24.995326    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6"
Sep 11 17:43:25.278137 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:25.278046    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6"]
Sep 11 17:43:25.282262 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:43:25.282231    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ef809a_4b68_4d84_8f0c_5585ee816aa0.slice/crio-c7196cf76c884066d8174d1d1ed2c17af416e2f6adfc85370bbaf9ef079b378a WatchSource:0}: Error finding container c7196cf76c884066d8174d1d1ed2c17af416e2f6adfc85370bbaf9ef079b378a: Status 404 returned error can't find the container with id c7196cf76c884066d8174d1d1ed2c17af416e2f6adfc85370bbaf9ef079b378a
Sep 11 17:43:25.470272 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:25.470232    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6" event={"ID":"33ef809a-4b68-4d84-8f0c-5585ee816aa0","Type":"ContainerStarted","Data":"c7196cf76c884066d8174d1d1ed2c17af416e2f6adfc85370bbaf9ef079b378a"}
Sep 11 17:43:25.626433 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:25.626404    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:26.477791 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:26.477752    2560 generic.go:356] "Generic (PLEG): container finished" podID="33ef809a-4b68-4d84-8f0c-5585ee816aa0" containerID="29a555459f12fd9c0ac25088012144dca2e8a69558549da61369eb62c49910bd" exitCode=0
Sep 11 17:43:26.478238 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:26.477832    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6" event={"ID":"33ef809a-4b68-4d84-8f0c-5585ee816aa0","Type":"ContainerDied","Data":"29a555459f12fd9c0ac25088012144dca2e8a69558549da61369eb62c49910bd"}
Sep 11 17:43:26.813632 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:26.813534    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:27.623367 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:27.623335    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6"
Sep 11 17:43:27.694435 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:27.694404    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ef809a-4b68-4d84-8f0c-5585ee816aa0-maas-api-service-ca\") pod \"33ef809a-4b68-4d84-8f0c-5585ee816aa0\" (UID: \"33ef809a-4b68-4d84-8f0c-5585ee816aa0\") "
Sep 11 17:43:27.694864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:27.694834    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ef809a-4b68-4d84-8f0c-5585ee816aa0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33ef809a-4b68-4d84-8f0c-5585ee816aa0" (UID: "33ef809a-4b68-4d84-8f0c-5585ee816aa0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:43:27.796154 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:27.796057    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33ef809a-4b68-4d84-8f0c-5585ee816aa0-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:43:28.057291 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:28.057207    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:28.490958 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:28.490926    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6"
Sep 11 17:43:28.490958 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:28.490934    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6" event={"ID":"33ef809a-4b68-4d84-8f0c-5585ee816aa0","Type":"ContainerDied","Data":"c7196cf76c884066d8174d1d1ed2c17af416e2f6adfc85370bbaf9ef079b378a"}
Sep 11 17:43:28.491227 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:28.490976    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7196cf76c884066d8174d1d1ed2c17af416e2f6adfc85370bbaf9ef079b378a"
Sep 11 17:43:29.250386 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:29.250356    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:30.281248 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:30.281211    2560 scope.go:117] "RemoveContainer" containerID="577215d4ddeaa818c03e38669656420c26ffd03e96364a7f571b7ccfe4f135fa"
Sep 11 17:43:30.388440 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:30.388408    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:31.744992 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:31.744956    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:33.338299 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:33.338266    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:35.057540 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:35.057506    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:41.470929 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:41.470890    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:41.639786 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:41.639754    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:41.765245 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:41.765165    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:41.974038 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:41.974005    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:42.094714 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:42.094627    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6"]
Sep 11 17:43:42.100658 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:42.100626    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-40f1fc-qc8j6"]
Sep 11 17:43:42.159739 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:42.159657    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:42.407686 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:42.407648    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:42.571712 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:42.571662    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:43.593696 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:43.593655    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:44.052112 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:44.052069    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33ef809a-4b68-4d84-8f0c-5585ee816aa0" path="/var/lib/kubelet/pods/33ef809a-4b68-4d84-8f0c-5585ee816aa0/volumes"
Sep 11 17:43:44.868582 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:44.868551    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:46.006538 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:46.006505    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:47.272993 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:47.272952    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:48.305720 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:48.305683    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:49.269569 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:49.269535    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:50.530538 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:50.530458    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:50.674367 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:50.674315    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:50.776186 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:50.776151    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"]
Sep 11 17:43:50.776398 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:50.776374    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s" podUID="b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" containerName="istio-proxy" containerID="cri-o://ddba8ea7ff13b36962ebdca7f78d8ef30eb75cfe87b472255f1152e644e37108" gracePeriod=30
Sep 11 17:43:50.795723 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:50.795621    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:51.141350 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:51.141310    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86db1c-59tj7"]
Sep 11 17:43:51.141745 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:51.141732    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ef809a-4b68-4d84-8f0c-5585ee816aa0" containerName="revoke-keys"
Sep 11 17:43:51.141799 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:51.141747    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ef809a-4b68-4d84-8f0c-5585ee816aa0" containerName="revoke-keys"
Sep 11 17:43:51.141836 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:51.141827    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="33ef809a-4b68-4d84-8f0c-5585ee816aa0" containerName="revoke-keys"
Sep 11 17:43:51.146296 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:51.146276    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86db1c-59tj7"
Sep 11 17:43:51.148985 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:51.148965    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:43:51.149089 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:51.149067    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:43:51.153676 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:51.153648    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86db1c-59tj7"]
Sep 11 17:43:51.234488 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:51.234437    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86602dd-28b9-4d55-978d-2574424496c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-86db1c-59tj7\" (UID: \"a86602dd-28b9-4d55-978d-2574424496c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86db1c-59tj7"
Sep 11 17:43:51.335907 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:51.335870    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86602dd-28b9-4d55-978d-2574424496c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-86db1c-59tj7\" (UID: \"a86602dd-28b9-4d55-978d-2574424496c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86db1c-59tj7"
Sep 11 17:43:51.336541 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:51.336522    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86602dd-28b9-4d55-978d-2574424496c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-86db1c-59tj7\" (UID: \"a86602dd-28b9-4d55-978d-2574424496c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86db1c-59tj7"
Sep 11 17:43:51.459373 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:51.459283    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86db1c-59tj7"
Sep 11 17:43:51.472772 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:51.472734    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:51.731898 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:51.731869    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86db1c-59tj7"]
Sep 11 17:43:51.733912 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:43:51.733881    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86602dd_28b9_4d55_978d_2574424496c7.slice/crio-0ff5d78477a04e42fd11619f86759a8321586949f71a0067b20060a369d9799b WatchSource:0}: Error finding container 0ff5d78477a04e42fd11619f86759a8321586949f71a0067b20060a369d9799b: Status 404 returned error can't find the container with id 0ff5d78477a04e42fd11619f86759a8321586949f71a0067b20060a369d9799b
Sep 11 17:43:52.377866 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:52.377833    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:52.635357 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:52.635329    2560 generic.go:356] "Generic (PLEG): container finished" podID="a86602dd-28b9-4d55-978d-2574424496c7" containerID="c100da3759dc6ad995e2cc6c368d113d31d9e59e163a4655cb6a15aa17380d16" exitCode=0
Sep 11 17:43:52.635544 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:52.635371    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86db1c-59tj7" event={"ID":"a86602dd-28b9-4d55-978d-2574424496c7","Type":"ContainerDied","Data":"c100da3759dc6ad995e2cc6c368d113d31d9e59e163a4655cb6a15aa17380d16"}
Sep 11 17:43:52.635544 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:52.635412    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86db1c-59tj7" event={"ID":"a86602dd-28b9-4d55-978d-2574424496c7","Type":"ContainerStarted","Data":"0ff5d78477a04e42fd11619f86759a8321586949f71a0067b20060a369d9799b"}
Sep 11 17:43:53.275941 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:53.275912    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:53.783331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:53.783305    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86db1c-59tj7"
Sep 11 17:43:53.862948 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:53.862903    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86602dd-28b9-4d55-978d-2574424496c7-maas-api-service-ca\") pod \"a86602dd-28b9-4d55-978d-2574424496c7\" (UID: \"a86602dd-28b9-4d55-978d-2574424496c7\") "
Sep 11 17:43:53.863365 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:53.863339    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a86602dd-28b9-4d55-978d-2574424496c7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a86602dd-28b9-4d55-978d-2574424496c7" (UID: "a86602dd-28b9-4d55-978d-2574424496c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:43:53.964295 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:53.964185    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86602dd-28b9-4d55-978d-2574424496c7-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:43:54.144942 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:54.144913    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:54.647836 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:54.647805    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86db1c-59tj7"
Sep 11 17:43:54.647836 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:54.647816    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86db1c-59tj7" event={"ID":"a86602dd-28b9-4d55-978d-2574424496c7","Type":"ContainerDied","Data":"0ff5d78477a04e42fd11619f86759a8321586949f71a0067b20060a369d9799b"}
Sep 11 17:43:54.648280 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:54.647856    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ff5d78477a04e42fd11619f86759a8321586949f71a0067b20060a369d9799b"
Sep 11 17:43:54.926142 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:54.926054    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:55.676109 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:55.676076    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:56.043042 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.043009    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:43:56.087407 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.087373    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-data\") pod \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") "
Sep 11 17:43:56.087605 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.087413    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-credential-socket\") pod \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") "
Sep 11 17:43:56.087605 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.087438    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-token\") pod \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") "
Sep 11 17:43:56.087605 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.087543    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-workload-certs\") pod \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") "
Sep 11 17:43:56.087778 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.087621    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-envoy\") pod \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") "
Sep 11 17:43:56.087778 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.087678    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istiod-ca-cert\") pod \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") "
Sep 11 17:43:56.087778 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.087739    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-podinfo\") pod \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") "
Sep 11 17:43:56.087778 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.087743    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" (UID: "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:43:56.087778 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.087762    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-data" (OuterVolumeSpecName: "istio-data") pod "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" (UID: "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:43:56.088038 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.087795    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-workload-socket\") pod \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") "
Sep 11 17:43:56.088038 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.087844    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zv2m\" (UniqueName: \"kubernetes.io/projected/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-kube-api-access-8zv2m\") pod \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\" (UID: \"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be\") "
Sep 11 17:43:56.088038 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.087918    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" (UID: "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:43:56.088310 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.088246    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-data\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:43:56.088310 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.088262    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" (UID: "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:43:56.088310 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.088269    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" (UID: "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:43:56.088310 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.088269    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-credential-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:43:56.088310 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.088310    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-workload-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:43:56.090154 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.090125    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-kube-api-access-8zv2m" (OuterVolumeSpecName: "kube-api-access-8zv2m") pod "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" (UID: "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be"). InnerVolumeSpecName "kube-api-access-8zv2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:43:56.090405 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.090381    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-token" (OuterVolumeSpecName: "istio-token") pod "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" (UID: "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:43:56.090504 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.090423    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" (UID: "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:43:56.090504 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.090432    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" (UID: "b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:43:56.190001 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.189959    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-envoy\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:43:56.190001 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.189998    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istiod-ca-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:43:56.190001 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.190010    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-podinfo\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:43:56.190250 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.190021    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-workload-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:43:56.190250 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.190035    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zv2m\" (UniqueName: \"kubernetes.io/projected/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-kube-api-access-8zv2m\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:43:56.190250 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.190049    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be-istio-token\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:43:56.660435 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.660401    2560 generic.go:356] "Generic (PLEG): container finished" podID="b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" containerID="ddba8ea7ff13b36962ebdca7f78d8ef30eb75cfe87b472255f1152e644e37108" exitCode=0
Sep 11 17:43:56.660639 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.660450    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"
Sep 11 17:43:56.660639 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.660507    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s" event={"ID":"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be","Type":"ContainerDied","Data":"ddba8ea7ff13b36962ebdca7f78d8ef30eb75cfe87b472255f1152e644e37108"}
Sep 11 17:43:56.660639 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.660564    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s" event={"ID":"b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be","Type":"ContainerDied","Data":"c3cb15ea23b9ac0ea0b9aeec13370923b9a1fde0ad32c8a2e5241e97aa9ef1ac"}
Sep 11 17:43:56.660639 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.660586    2560 scope.go:117] "RemoveContainer" containerID="ddba8ea7ff13b36962ebdca7f78d8ef30eb75cfe87b472255f1152e644e37108"
Sep 11 17:43:56.670216 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.670195    2560 scope.go:117] "RemoveContainer" containerID="d110fbb2a9fcd4a79a7289a11c00931e416b8e411fb0d1923e9f2462689d319f"
Sep 11 17:43:56.679291 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.679262    2560 scope.go:117] "RemoveContainer" containerID="ddba8ea7ff13b36962ebdca7f78d8ef30eb75cfe87b472255f1152e644e37108"
Sep 11 17:43:56.679632 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:43:56.679612    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddba8ea7ff13b36962ebdca7f78d8ef30eb75cfe87b472255f1152e644e37108\": container with ID starting with ddba8ea7ff13b36962ebdca7f78d8ef30eb75cfe87b472255f1152e644e37108 not found: ID does not exist" containerID="ddba8ea7ff13b36962ebdca7f78d8ef30eb75cfe87b472255f1152e644e37108"
Sep 11 17:43:56.679678 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.679643    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddba8ea7ff13b36962ebdca7f78d8ef30eb75cfe87b472255f1152e644e37108"} err="failed to get container status \"ddba8ea7ff13b36962ebdca7f78d8ef30eb75cfe87b472255f1152e644e37108\": rpc error: code = NotFound desc = could not find container \"ddba8ea7ff13b36962ebdca7f78d8ef30eb75cfe87b472255f1152e644e37108\": container with ID starting with ddba8ea7ff13b36962ebdca7f78d8ef30eb75cfe87b472255f1152e644e37108 not found: ID does not exist"
Sep 11 17:43:56.679678 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.679663    2560 scope.go:117] "RemoveContainer" containerID="d110fbb2a9fcd4a79a7289a11c00931e416b8e411fb0d1923e9f2462689d319f"
Sep 11 17:43:56.679949 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:43:56.679932    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d110fbb2a9fcd4a79a7289a11c00931e416b8e411fb0d1923e9f2462689d319f\": container with ID starting with d110fbb2a9fcd4a79a7289a11c00931e416b8e411fb0d1923e9f2462689d319f not found: ID does not exist" containerID="d110fbb2a9fcd4a79a7289a11c00931e416b8e411fb0d1923e9f2462689d319f"
Sep 11 17:43:56.679987 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.679959    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d110fbb2a9fcd4a79a7289a11c00931e416b8e411fb0d1923e9f2462689d319f"} err="failed to get container status \"d110fbb2a9fcd4a79a7289a11c00931e416b8e411fb0d1923e9f2462689d319f\": rpc error: code = NotFound desc = could not find container \"d110fbb2a9fcd4a79a7289a11c00931e416b8e411fb0d1923e9f2462689d319f\": container with ID starting with d110fbb2a9fcd4a79a7289a11c00931e416b8e411fb0d1923e9f2462689d319f not found: ID does not exist"
Sep 11 17:43:56.689135 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.689103    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"]
Sep 11 17:43:56.694555 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.694528    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:43:56.695071 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:56.695052    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-40f1fc-openshift-default-85bff965c6-mb54s"]
Sep 11 17:43:58.051329 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:58.051296    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" path="/var/lib/kubelet/pods/b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be/volumes"
Sep 11 17:43:59.697725 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:43:59.697683    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:00.632286 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:00.632249    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:06.406815 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.406762    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:06.572327 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.572294    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:06.736357 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.736238    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:06.796565 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.796524    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86db1c-59tj7"]
Sep 11 17:44:06.799685 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.799652    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86db1c-59tj7"]
Sep 11 17:44:06.824456 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.824421    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:06.974744 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.974708    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7729-f6a310"]
Sep 11 17:44:06.975138 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.975125    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" containerName="istio-proxy"
Sep 11 17:44:06.975200 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.975141    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" containerName="istio-proxy"
Sep 11 17:44:06.975200 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.975150    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a86602dd-28b9-4d55-978d-2574424496c7" containerName="revoke-keys"
Sep 11 17:44:06.975200 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.975155    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a86602dd-28b9-4d55-978d-2574424496c7" containerName="revoke-keys"
Sep 11 17:44:06.975296 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.975253    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a86602dd-28b9-4d55-978d-2574424496c7" containerName="revoke-keys"
Sep 11 17:44:06.975296 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.975264    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" containerName="istio-proxy"
Sep 11 17:44:06.978209 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.978193    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-f6a310"
Sep 11 17:44:06.983488 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.983456    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jtt66\""
Sep 11 17:44:06.989683 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:06.989625    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7729-f6a310"]
Sep 11 17:44:07.108116 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:07.108077    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6bb\" (UniqueName: \"kubernetes.io/projected/3b10a0e8-d576-4f13-9bd6-e315364d330a-kube-api-access-7l6bb\") pod \"test-curl-7729-f6a310\" (UID: \"3b10a0e8-d576-4f13-9bd6-e315364d330a\") " pod="opendatahub/test-curl-7729-f6a310"
Sep 11 17:44:07.209547 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:07.209509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6bb\" (UniqueName: \"kubernetes.io/projected/3b10a0e8-d576-4f13-9bd6-e315364d330a-kube-api-access-7l6bb\") pod \"test-curl-7729-f6a310\" (UID: \"3b10a0e8-d576-4f13-9bd6-e315364d330a\") " pod="opendatahub/test-curl-7729-f6a310"
Sep 11 17:44:07.218818 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:07.218776    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:07.236522 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:07.236495    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6bb\" (UniqueName: \"kubernetes.io/projected/3b10a0e8-d576-4f13-9bd6-e315364d330a-kube-api-access-7l6bb\") pod \"test-curl-7729-f6a310\" (UID: \"3b10a0e8-d576-4f13-9bd6-e315364d330a\") " pod="opendatahub/test-curl-7729-f6a310"
Sep 11 17:44:07.287833 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:07.287804    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-f6a310"
Sep 11 17:44:07.413331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:07.413296    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:07.570371 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:07.570346    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7729-f6a310"]
Sep 11 17:44:07.572400 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:44:07.572369    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b10a0e8_d576_4f13_9bd6_e315364d330a.slice/crio-1ec99b13c9083174defe0f243242b2de8f4831da6be1a0497e009b8b20d80efe WatchSource:0}: Error finding container 1ec99b13c9083174defe0f243242b2de8f4831da6be1a0497e009b8b20d80efe: Status 404 returned error can't find the container with id 1ec99b13c9083174defe0f243242b2de8f4831da6be1a0497e009b8b20d80efe
Sep 11 17:44:07.734521 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:07.734467    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7729-f6a310" event={"ID":"3b10a0e8-d576-4f13-9bd6-e315364d330a","Type":"ContainerStarted","Data":"e2935ff9b19cedce9de359f10320b12401bb750eb76f611a5e59bb414f6a82a1"}
Sep 11 17:44:07.734714 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:07.734531    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7729-f6a310" event={"ID":"3b10a0e8-d576-4f13-9bd6-e315364d330a","Type":"ContainerStarted","Data":"1ec99b13c9083174defe0f243242b2de8f4831da6be1a0497e009b8b20d80efe"}
Sep 11 17:44:07.756097 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:07.756047    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7729-f6a310" podStartSLOduration=1.75603072 podStartE2EDuration="1.75603072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:44:07.755957336 +0000 UTC m=+1418.188410113" watchObservedRunningTime="2026-09-11 17:44:07.75603072 +0000 UTC m=+1418.188483497"
Sep 11 17:44:08.051876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:08.051842    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a86602dd-28b9-4d55-978d-2574424496c7" path="/var/lib/kubelet/pods/a86602dd-28b9-4d55-978d-2574424496c7/volumes"
Sep 11 17:44:08.395655 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:08.395616    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7729-f6a310"]
Sep 11 17:44:08.399130 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:08.399103    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7729-f6a310"]
Sep 11 17:44:08.426010 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:08.425893    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:08.739867 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:44:08.739782    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7729-f6a310" container="test-curl-7729-f6a310"
Sep 11 17:44:09.358201 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:09.358169    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:09.746299 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:09.746254    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7729-f6a310" podUID="3b10a0e8-d576-4f13-9bd6-e315364d330a" containerName="test-curl-7729-f6a310" containerID="cri-o://e2935ff9b19cedce9de359f10320b12401bb750eb76f611a5e59bb414f6a82a1" gracePeriod=2
Sep 11 17:44:09.749039 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:44:09.749004    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7729-f6a310\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="3b10a0e8-d576-4f13-9bd6-e315364d330a" pod="opendatahub/test-curl-7729-f6a310"
Sep 11 17:44:10.049743 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:44:10.049647    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7729-f6a310\" is forbidden: User \"system:node:ip-10-0-138-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-148.ec2.internal' and this object" podUID="3b10a0e8-d576-4f13-9bd6-e315364d330a" pod="opendatahub/test-curl-7729-f6a310"
Sep 11 17:44:10.197638 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:10.197587    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:11.177045 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:11.177011    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:11.937074 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:11.937048    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-f6a310"
Sep 11 17:44:12.060395 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:12.060298    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l6bb\" (UniqueName: \"kubernetes.io/projected/3b10a0e8-d576-4f13-9bd6-e315364d330a-kube-api-access-7l6bb\") pod \"3b10a0e8-d576-4f13-9bd6-e315364d330a\" (UID: \"3b10a0e8-d576-4f13-9bd6-e315364d330a\") "
Sep 11 17:44:12.062367 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:12.062342    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b10a0e8-d576-4f13-9bd6-e315364d330a-kube-api-access-7l6bb" (OuterVolumeSpecName: "kube-api-access-7l6bb") pod "3b10a0e8-d576-4f13-9bd6-e315364d330a" (UID: "3b10a0e8-d576-4f13-9bd6-e315364d330a"). InnerVolumeSpecName "kube-api-access-7l6bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:12.162297 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:12.162264    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l6bb\" (UniqueName: \"kubernetes.io/projected/3b10a0e8-d576-4f13-9bd6-e315364d330a-kube-api-access-7l6bb\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:12.722332 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:12.722285    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"]
Sep 11 17:44:12.734982 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:12.734939    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:12.763508 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:12.763448    2560 generic.go:356] "Generic (PLEG): container finished" podID="3b10a0e8-d576-4f13-9bd6-e315364d330a" containerID="e2935ff9b19cedce9de359f10320b12401bb750eb76f611a5e59bb414f6a82a1" exitCode=137
Sep 11 17:44:12.763508 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:12.763505    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7729-f6a310"
Sep 11 17:44:12.763766 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:12.763529    2560 scope.go:117] "RemoveContainer" containerID="e2935ff9b19cedce9de359f10320b12401bb750eb76f611a5e59bb414f6a82a1"
Sep 11 17:44:12.775434 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:12.775413    2560 scope.go:117] "RemoveContainer" containerID="e2935ff9b19cedce9de359f10320b12401bb750eb76f611a5e59bb414f6a82a1"
Sep 11 17:44:12.775828 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:44:12.775804    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2935ff9b19cedce9de359f10320b12401bb750eb76f611a5e59bb414f6a82a1\": container with ID starting with e2935ff9b19cedce9de359f10320b12401bb750eb76f611a5e59bb414f6a82a1 not found: ID does not exist" containerID="e2935ff9b19cedce9de359f10320b12401bb750eb76f611a5e59bb414f6a82a1"
Sep 11 17:44:12.775897 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:12.775838    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2935ff9b19cedce9de359f10320b12401bb750eb76f611a5e59bb414f6a82a1"} err="failed to get container status \"e2935ff9b19cedce9de359f10320b12401bb750eb76f611a5e59bb414f6a82a1\": rpc error: code = NotFound desc = could not find container \"e2935ff9b19cedce9de359f10320b12401bb750eb76f611a5e59bb414f6a82a1\": container with ID starting with e2935ff9b19cedce9de359f10320b12401bb750eb76f611a5e59bb414f6a82a1 not found: ID does not exist"
Sep 11 17:44:13.759660 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:13.759623    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"]
Sep 11 17:44:13.767082 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:13.767054    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:13.861211 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:13.861138    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:14.051588 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.051506    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b10a0e8-d576-4f13-9bd6-e315364d330a" path="/var/lib/kubelet/pods/3b10a0e8-d576-4f13-9bd6-e315364d330a/volumes"
Sep 11 17:44:14.051811 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.051796    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"]
Sep 11 17:44:14.052037 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.052016    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64" podUID="7102f469-a6f9-4502-82d4-dcab6acbfda3" containerName="istio-proxy" containerID="cri-o://30330c81adb656c23007542c8197e9b2023e97862eacfd684d7f0ed3ee712f55" gracePeriod=30
Sep 11 17:44:14.103165 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.103126    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:14.368142 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.368047    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q"]
Sep 11 17:44:14.368933 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.368910    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" containerName="istio-proxy"
Sep 11 17:44:14.369066 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.369055    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" containerName="istio-proxy"
Sep 11 17:44:14.369154 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.369144    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b10a0e8-d576-4f13-9bd6-e315364d330a" containerName="test-curl-7729-f6a310"
Sep 11 17:44:14.369219 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.369211    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b10a0e8-d576-4f13-9bd6-e315364d330a" containerName="test-curl-7729-f6a310"
Sep 11 17:44:14.369387 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.369377    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b10a0e8-d576-4f13-9bd6-e315364d330a" containerName="test-curl-7729-f6a310"
Sep 11 17:44:14.369458 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.369448    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b50bf805-a6c5-4fc8-8dd4-2cbedee6a4be" containerName="istio-proxy"
Sep 11 17:44:14.374676 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.374649    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q"
Sep 11 17:44:14.379768 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.379732    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:44:14.380224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.380204    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:44:14.388041 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.388018    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q"]
Sep 11 17:44:14.487002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.486957    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3614e2-eb68-4368-abff-ce6ae5ac5347-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q\" (UID: \"ab3614e2-eb68-4368-abff-ce6ae5ac5347\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q"
Sep 11 17:44:14.588082 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.588040    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3614e2-eb68-4368-abff-ce6ae5ac5347-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q\" (UID: \"ab3614e2-eb68-4368-abff-ce6ae5ac5347\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q"
Sep 11 17:44:14.588724 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.588705    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3614e2-eb68-4368-abff-ce6ae5ac5347-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q\" (UID: \"ab3614e2-eb68-4368-abff-ce6ae5ac5347\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q"
Sep 11 17:44:14.697936 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.697896    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q"
Sep 11 17:44:14.769173 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:14.769091    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:15.003990 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:15.003954    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q"]
Sep 11 17:44:15.005292 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:44:15.005260    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3614e2_eb68_4368_abff_ce6ae5ac5347.slice/crio-95a52f60924af04f1e775f7f7bb1fa72a366864a6ca4fc19bcb878cc017d1b9b WatchSource:0}: Error finding container 95a52f60924af04f1e775f7f7bb1fa72a366864a6ca4fc19bcb878cc017d1b9b: Status 404 returned error can't find the container with id 95a52f60924af04f1e775f7f7bb1fa72a366864a6ca4fc19bcb878cc017d1b9b
Sep 11 17:44:15.784198 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:15.784163    2560 generic.go:356] "Generic (PLEG): container finished" podID="ab3614e2-eb68-4368-abff-ce6ae5ac5347" containerID="a64c51287d018cc49908f36120e97fc3ae602f49cb728ae86e09defe1cfa8d5c" exitCode=0
Sep 11 17:44:15.784647 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:15.784207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q" event={"ID":"ab3614e2-eb68-4368-abff-ce6ae5ac5347","Type":"ContainerDied","Data":"a64c51287d018cc49908f36120e97fc3ae602f49cb728ae86e09defe1cfa8d5c"}
Sep 11 17:44:15.784647 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:15.784293    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q" event={"ID":"ab3614e2-eb68-4368-abff-ce6ae5ac5347","Type":"ContainerStarted","Data":"95a52f60924af04f1e775f7f7bb1fa72a366864a6ca4fc19bcb878cc017d1b9b"}
Sep 11 17:44:15.877729 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:15.877694    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:16.892675 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:16.892639    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:16.924699 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:16.924673    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q"
Sep 11 17:44:17.116634 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:17.116541    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3614e2-eb68-4368-abff-ce6ae5ac5347-maas-api-service-ca\") pod \"ab3614e2-eb68-4368-abff-ce6ae5ac5347\" (UID: \"ab3614e2-eb68-4368-abff-ce6ae5ac5347\") "
Sep 11 17:44:17.116984 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:17.116953    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3614e2-eb68-4368-abff-ce6ae5ac5347-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ab3614e2-eb68-4368-abff-ce6ae5ac5347" (UID: "ab3614e2-eb68-4368-abff-ce6ae5ac5347"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:17.217522 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:17.217456    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3614e2-eb68-4368-abff-ce6ae5ac5347-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:17.797457 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:17.797429    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q"
Sep 11 17:44:17.797719 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:17.797469    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q" event={"ID":"ab3614e2-eb68-4368-abff-ce6ae5ac5347","Type":"ContainerDied","Data":"95a52f60924af04f1e775f7f7bb1fa72a366864a6ca4fc19bcb878cc017d1b9b"}
Sep 11 17:44:17.797719 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:17.797631    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95a52f60924af04f1e775f7f7bb1fa72a366864a6ca4fc19bcb878cc017d1b9b"
Sep 11 17:44:17.868408 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:17.868377    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:18.827456 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:18.827410    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:19.308586 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.308562    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:44:19.338241 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.338201    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-credential-socket\") pod \"7102f469-a6f9-4502-82d4-dcab6acbfda3\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") "
Sep 11 17:44:19.338399 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.338258    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-workload-socket\") pod \"7102f469-a6f9-4502-82d4-dcab6acbfda3\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") "
Sep 11 17:44:19.338399 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.338284    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-workload-certs\") pod \"7102f469-a6f9-4502-82d4-dcab6acbfda3\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") "
Sep 11 17:44:19.338399 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.338310    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-token\") pod \"7102f469-a6f9-4502-82d4-dcab6acbfda3\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") "
Sep 11 17:44:19.338399 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.338368    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-podinfo\") pod \"7102f469-a6f9-4502-82d4-dcab6acbfda3\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") "
Sep 11 17:44:19.338631 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.338415    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-data\") pod \"7102f469-a6f9-4502-82d4-dcab6acbfda3\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") "
Sep 11 17:44:19.338631 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.338450    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7102f469-a6f9-4502-82d4-dcab6acbfda3-istiod-ca-cert\") pod \"7102f469-a6f9-4502-82d4-dcab6acbfda3\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") "
Sep 11 17:44:19.338631 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.338545    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7102f469-a6f9-4502-82d4-dcab6acbfda3" (UID: "7102f469-a6f9-4502-82d4-dcab6acbfda3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:19.338770 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.338632    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-envoy\") pod \"7102f469-a6f9-4502-82d4-dcab6acbfda3\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") "
Sep 11 17:44:19.338770 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.338642    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7102f469-a6f9-4502-82d4-dcab6acbfda3" (UID: "7102f469-a6f9-4502-82d4-dcab6acbfda3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:19.338770 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.338677    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwnwg\" (UniqueName: \"kubernetes.io/projected/7102f469-a6f9-4502-82d4-dcab6acbfda3-kube-api-access-wwnwg\") pod \"7102f469-a6f9-4502-82d4-dcab6acbfda3\" (UID: \"7102f469-a6f9-4502-82d4-dcab6acbfda3\") "
Sep 11 17:44:19.338770 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.338683    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7102f469-a6f9-4502-82d4-dcab6acbfda3" (UID: "7102f469-a6f9-4502-82d4-dcab6acbfda3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:19.338987 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.338789    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-data" (OuterVolumeSpecName: "istio-data") pod "7102f469-a6f9-4502-82d4-dcab6acbfda3" (UID: "7102f469-a6f9-4502-82d4-dcab6acbfda3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:19.338987 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.338888    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7102f469-a6f9-4502-82d4-dcab6acbfda3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7102f469-a6f9-4502-82d4-dcab6acbfda3" (UID: "7102f469-a6f9-4502-82d4-dcab6acbfda3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:19.339148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.339124    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-workload-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:19.339224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.339152    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-workload-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:19.339224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.339167    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-data\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:19.339224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.339180    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7102f469-a6f9-4502-82d4-dcab6acbfda3-istiod-ca-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:19.339224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.339192    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-credential-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:19.340846 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.340818    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-token" (OuterVolumeSpecName: "istio-token") pod "7102f469-a6f9-4502-82d4-dcab6acbfda3" (UID: "7102f469-a6f9-4502-82d4-dcab6acbfda3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:19.340846 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.340837    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7102f469-a6f9-4502-82d4-dcab6acbfda3" (UID: "7102f469-a6f9-4502-82d4-dcab6acbfda3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:19.341031 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.340863    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7102f469-a6f9-4502-82d4-dcab6acbfda3" (UID: "7102f469-a6f9-4502-82d4-dcab6acbfda3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:44:19.341031 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.340863    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7102f469-a6f9-4502-82d4-dcab6acbfda3-kube-api-access-wwnwg" (OuterVolumeSpecName: "kube-api-access-wwnwg") pod "7102f469-a6f9-4502-82d4-dcab6acbfda3" (UID: "7102f469-a6f9-4502-82d4-dcab6acbfda3"). InnerVolumeSpecName "kube-api-access-wwnwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:19.440071 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.439986    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-token\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:19.440071 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.440013    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-podinfo\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:19.440071 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.440022    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7102f469-a6f9-4502-82d4-dcab6acbfda3-istio-envoy\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:19.440071 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.440031    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwnwg\" (UniqueName: \"kubernetes.io/projected/7102f469-a6f9-4502-82d4-dcab6acbfda3-kube-api-access-wwnwg\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:19.810993 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.810907    2560 generic.go:356] "Generic (PLEG): container finished" podID="7102f469-a6f9-4502-82d4-dcab6acbfda3" containerID="30330c81adb656c23007542c8197e9b2023e97862eacfd684d7f0ed3ee712f55" exitCode=0
Sep 11 17:44:19.810993 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.810965    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"
Sep 11 17:44:19.811181 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.810998    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64" event={"ID":"7102f469-a6f9-4502-82d4-dcab6acbfda3","Type":"ContainerDied","Data":"30330c81adb656c23007542c8197e9b2023e97862eacfd684d7f0ed3ee712f55"}
Sep 11 17:44:19.811181 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.811056    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64" event={"ID":"7102f469-a6f9-4502-82d4-dcab6acbfda3","Type":"ContainerDied","Data":"29dee3d25ac540c49e6838adcf0c922e9fa5156e88d711992fbcbfbe0ed2f662"}
Sep 11 17:44:19.811181 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.811087    2560 scope.go:117] "RemoveContainer" containerID="30330c81adb656c23007542c8197e9b2023e97862eacfd684d7f0ed3ee712f55"
Sep 11 17:44:19.820855 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.820834    2560 scope.go:117] "RemoveContainer" containerID="53bf18523fa0b96e18358e65239c32e8b0bdb99e05d4c3ef2541d13bb1e86f33"
Sep 11 17:44:19.828893 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.828876    2560 scope.go:117] "RemoveContainer" containerID="30330c81adb656c23007542c8197e9b2023e97862eacfd684d7f0ed3ee712f55"
Sep 11 17:44:19.829162 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:44:19.829140    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30330c81adb656c23007542c8197e9b2023e97862eacfd684d7f0ed3ee712f55\": container with ID starting with 30330c81adb656c23007542c8197e9b2023e97862eacfd684d7f0ed3ee712f55 not found: ID does not exist" containerID="30330c81adb656c23007542c8197e9b2023e97862eacfd684d7f0ed3ee712f55"
Sep 11 17:44:19.829200 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.829170    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30330c81adb656c23007542c8197e9b2023e97862eacfd684d7f0ed3ee712f55"} err="failed to get container status \"30330c81adb656c23007542c8197e9b2023e97862eacfd684d7f0ed3ee712f55\": rpc error: code = NotFound desc = could not find container \"30330c81adb656c23007542c8197e9b2023e97862eacfd684d7f0ed3ee712f55\": container with ID starting with 30330c81adb656c23007542c8197e9b2023e97862eacfd684d7f0ed3ee712f55 not found: ID does not exist"
Sep 11 17:44:19.829200 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.829186    2560 scope.go:117] "RemoveContainer" containerID="53bf18523fa0b96e18358e65239c32e8b0bdb99e05d4c3ef2541d13bb1e86f33"
Sep 11 17:44:19.829415 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:44:19.829397    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53bf18523fa0b96e18358e65239c32e8b0bdb99e05d4c3ef2541d13bb1e86f33\": container with ID starting with 53bf18523fa0b96e18358e65239c32e8b0bdb99e05d4c3ef2541d13bb1e86f33 not found: ID does not exist" containerID="53bf18523fa0b96e18358e65239c32e8b0bdb99e05d4c3ef2541d13bb1e86f33"
Sep 11 17:44:19.829449 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.829421    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53bf18523fa0b96e18358e65239c32e8b0bdb99e05d4c3ef2541d13bb1e86f33"} err="failed to get container status \"53bf18523fa0b96e18358e65239c32e8b0bdb99e05d4c3ef2541d13bb1e86f33\": rpc error: code = NotFound desc = could not find container \"53bf18523fa0b96e18358e65239c32e8b0bdb99e05d4c3ef2541d13bb1e86f33\": container with ID starting with 53bf18523fa0b96e18358e65239c32e8b0bdb99e05d4c3ef2541d13bb1e86f33 not found: ID does not exist"
Sep 11 17:44:19.848966 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.848940    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"]
Sep 11 17:44:19.866686 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:19.866660    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-86db1c-openshift-default-6d57f5f86b-n4g64"]
Sep 11 17:44:20.051466 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:20.051434    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7102f469-a6f9-4502-82d4-dcab6acbfda3" path="/var/lib/kubelet/pods/7102f469-a6f9-4502-82d4-dcab6acbfda3/volumes"
Sep 11 17:44:20.213618 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:20.213579    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:25.151244 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:25.151164    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:25.316792 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:25.316754    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:25.557876 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:25.557758    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:25.585582 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:25.585545    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q"]
Sep 11 17:44:25.593685 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:25.593428    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-470521-m2j2q"]
Sep 11 17:44:25.719459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:25.719427    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:26.053347 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:26.053300    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab3614e2-eb68-4368-abff-ce6ae5ac5347" path="/var/lib/kubelet/pods/ab3614e2-eb68-4368-abff-ce6ae5ac5347/volumes"
Sep 11 17:44:26.076465 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:26.076424    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:26.238939 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:26.238901    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:27.144158 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:27.144118    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:27.741831 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:27.741793    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v" podUID="434404c1-2a0e-4c82-ad92-05f53eb21111" containerName="main" containerID="cri-o://fd9aaa9c9b06e90a6748be595713289ce83567513facb1bc0f3b88fd3ae94ad3" gracePeriod=45
Sep 11 17:44:27.857005 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:27.856973    2560 generic.go:356] "Generic (PLEG): container finished" podID="434404c1-2a0e-4c82-ad92-05f53eb21111" containerID="fd9aaa9c9b06e90a6748be595713289ce83567513facb1bc0f3b88fd3ae94ad3" exitCode=0
Sep 11 17:44:27.857158 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:27.857044    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v" event={"ID":"434404c1-2a0e-4c82-ad92-05f53eb21111","Type":"ContainerDied","Data":"fd9aaa9c9b06e90a6748be595713289ce83567513facb1bc0f3b88fd3ae94ad3"}
Sep 11 17:44:27.986332 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:27.986306    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:44:28.030017 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.029938    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/434404c1-2a0e-4c82-ad92-05f53eb21111-tls-certs\") pod \"434404c1-2a0e-4c82-ad92-05f53eb21111\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") "
Sep 11 17:44:28.030017 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.030012    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-tmp-dir\") pod \"434404c1-2a0e-4c82-ad92-05f53eb21111\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") "
Sep 11 17:44:28.030237 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.030048    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-home\") pod \"434404c1-2a0e-4c82-ad92-05f53eb21111\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") "
Sep 11 17:44:28.030237 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.030084    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-model-cache\") pod \"434404c1-2a0e-4c82-ad92-05f53eb21111\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") "
Sep 11 17:44:28.030237 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.030120    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z858b\" (UniqueName: \"kubernetes.io/projected/434404c1-2a0e-4c82-ad92-05f53eb21111-kube-api-access-z858b\") pod \"434404c1-2a0e-4c82-ad92-05f53eb21111\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") "
Sep 11 17:44:28.030237 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.030156    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-dshm\") pod \"434404c1-2a0e-4c82-ad92-05f53eb21111\" (UID: \"434404c1-2a0e-4c82-ad92-05f53eb21111\") "
Sep 11 17:44:28.030441 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.030365    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-home" (OuterVolumeSpecName: "home") pod "434404c1-2a0e-4c82-ad92-05f53eb21111" (UID: "434404c1-2a0e-4c82-ad92-05f53eb21111"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:28.030441 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.030363    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "434404c1-2a0e-4c82-ad92-05f53eb21111" (UID: "434404c1-2a0e-4c82-ad92-05f53eb21111"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:28.030441 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.030391    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-model-cache" (OuterVolumeSpecName: "model-cache") pod "434404c1-2a0e-4c82-ad92-05f53eb21111" (UID: "434404c1-2a0e-4c82-ad92-05f53eb21111"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:28.030616 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.030543    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:28.030616 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.030564    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:28.030616 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.030577    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:28.032256 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.032230    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-dshm" (OuterVolumeSpecName: "dshm") pod "434404c1-2a0e-4c82-ad92-05f53eb21111" (UID: "434404c1-2a0e-4c82-ad92-05f53eb21111"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:28.032675 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.032653    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434404c1-2a0e-4c82-ad92-05f53eb21111-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "434404c1-2a0e-4c82-ad92-05f53eb21111" (UID: "434404c1-2a0e-4c82-ad92-05f53eb21111"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:44:28.032757 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.032652    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434404c1-2a0e-4c82-ad92-05f53eb21111-kube-api-access-z858b" (OuterVolumeSpecName: "kube-api-access-z858b") pod "434404c1-2a0e-4c82-ad92-05f53eb21111" (UID: "434404c1-2a0e-4c82-ad92-05f53eb21111"). InnerVolumeSpecName "kube-api-access-z858b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:28.132034 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.131993    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z858b\" (UniqueName: \"kubernetes.io/projected/434404c1-2a0e-4c82-ad92-05f53eb21111-kube-api-access-z858b\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:28.132034 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.132030    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/434404c1-2a0e-4c82-ad92-05f53eb21111-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:28.132034 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.132040    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/434404c1-2a0e-4c82-ad92-05f53eb21111-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:28.466108 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.466069    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:28.779661 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.779549    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw" podUID="7ee86935-b26d-4e03-8af2-ba3c8ac78055" containerName="main" containerID="cri-o://2aa17915778cd6c0caf6fba3d56d71d521725da6a70d39cbcf820878b9f8c966" gracePeriod=45
Sep 11 17:44:28.863841 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.863807    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"
Sep 11 17:44:28.863841 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.863835    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v" event={"ID":"434404c1-2a0e-4c82-ad92-05f53eb21111","Type":"ContainerDied","Data":"72e4182bc9933cf851294b1ac7ac06576dc46244d1bb579b075699b61c33d40e"}
Sep 11 17:44:28.864106 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.863882    2560 scope.go:117] "RemoveContainer" containerID="fd9aaa9c9b06e90a6748be595713289ce83567513facb1bc0f3b88fd3ae94ad3"
Sep 11 17:44:28.886965 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.886933    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"]
Sep 11 17:44:28.898466 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:28.898447    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/sub-test-model-4c36f4-kserve-8474569579-qgh4v"]
Sep 11 17:44:29.045853 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.045781    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:44:29.143133 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.143097    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee86935-b26d-4e03-8af2-ba3c8ac78055-tls-certs\") pod \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") "
Sep 11 17:44:29.143331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.143158    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-tmp-dir\") pod \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") "
Sep 11 17:44:29.143331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.143213    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-home\") pod \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") "
Sep 11 17:44:29.143331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.143248    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-dshm\") pod \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") "
Sep 11 17:44:29.143331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.143283    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q74d7\" (UniqueName: \"kubernetes.io/projected/7ee86935-b26d-4e03-8af2-ba3c8ac78055-kube-api-access-q74d7\") pod \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") "
Sep 11 17:44:29.143331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.143309    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-model-cache\") pod \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\" (UID: \"7ee86935-b26d-4e03-8af2-ba3c8ac78055\") "
Sep 11 17:44:29.143665 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.143468    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-home" (OuterVolumeSpecName: "home") pod "7ee86935-b26d-4e03-8af2-ba3c8ac78055" (UID: "7ee86935-b26d-4e03-8af2-ba3c8ac78055"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:29.143665 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.143519    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ee86935-b26d-4e03-8af2-ba3c8ac78055" (UID: "7ee86935-b26d-4e03-8af2-ba3c8ac78055"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:29.143777 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.143711    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ee86935-b26d-4e03-8af2-ba3c8ac78055" (UID: "7ee86935-b26d-4e03-8af2-ba3c8ac78055"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:29.144009 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.143985    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:29.144105 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.144013    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:29.144105 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.144028    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:29.145651 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.145625    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-dshm" (OuterVolumeSpecName: "dshm") pod "7ee86935-b26d-4e03-8af2-ba3c8ac78055" (UID: "7ee86935-b26d-4e03-8af2-ba3c8ac78055"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:29.145651 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.145637    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee86935-b26d-4e03-8af2-ba3c8ac78055-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ee86935-b26d-4e03-8af2-ba3c8ac78055" (UID: "7ee86935-b26d-4e03-8af2-ba3c8ac78055"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:44:29.145773 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.145659    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee86935-b26d-4e03-8af2-ba3c8ac78055-kube-api-access-q74d7" (OuterVolumeSpecName: "kube-api-access-q74d7") pod "7ee86935-b26d-4e03-8af2-ba3c8ac78055" (UID: "7ee86935-b26d-4e03-8af2-ba3c8ac78055"). InnerVolumeSpecName "kube-api-access-q74d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:29.245430 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.245388    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee86935-b26d-4e03-8af2-ba3c8ac78055-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:29.245430 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.245425    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q74d7\" (UniqueName: \"kubernetes.io/projected/7ee86935-b26d-4e03-8af2-ba3c8ac78055-kube-api-access-q74d7\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:29.245430 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.245435    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee86935-b26d-4e03-8af2-ba3c8ac78055-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:29.412201 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.412169    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:29.871782 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.871681    2560 generic.go:356] "Generic (PLEG): container finished" podID="7ee86935-b26d-4e03-8af2-ba3c8ac78055" containerID="2aa17915778cd6c0caf6fba3d56d71d521725da6a70d39cbcf820878b9f8c966" exitCode=0
Sep 11 17:44:29.871782 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.871719    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"
Sep 11 17:44:29.871782 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.871766    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw" event={"ID":"7ee86935-b26d-4e03-8af2-ba3c8ac78055","Type":"ContainerDied","Data":"2aa17915778cd6c0caf6fba3d56d71d521725da6a70d39cbcf820878b9f8c966"}
Sep 11 17:44:29.872372 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.871807    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw" event={"ID":"7ee86935-b26d-4e03-8af2-ba3c8ac78055","Type":"ContainerDied","Data":"9749565a74bba427329b511dceb89026ef831f4c5fb45b65a246671571cea400"}
Sep 11 17:44:29.872372 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.871829    2560 scope.go:117] "RemoveContainer" containerID="2aa17915778cd6c0caf6fba3d56d71d521725da6a70d39cbcf820878b9f8c966"
Sep 11 17:44:29.881355 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.881336    2560 scope.go:117] "RemoveContainer" containerID="2aa17915778cd6c0caf6fba3d56d71d521725da6a70d39cbcf820878b9f8c966"
Sep 11 17:44:29.881684 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:44:29.881661    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa17915778cd6c0caf6fba3d56d71d521725da6a70d39cbcf820878b9f8c966\": container with ID starting with 2aa17915778cd6c0caf6fba3d56d71d521725da6a70d39cbcf820878b9f8c966 not found: ID does not exist" containerID="2aa17915778cd6c0caf6fba3d56d71d521725da6a70d39cbcf820878b9f8c966"
Sep 11 17:44:29.881775 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.881689    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa17915778cd6c0caf6fba3d56d71d521725da6a70d39cbcf820878b9f8c966"} err="failed to get container status \"2aa17915778cd6c0caf6fba3d56d71d521725da6a70d39cbcf820878b9f8c966\": rpc error: code = NotFound desc = could not find container \"2aa17915778cd6c0caf6fba3d56d71d521725da6a70d39cbcf820878b9f8c966\": container with ID starting with 2aa17915778cd6c0caf6fba3d56d71d521725da6a70d39cbcf820878b9f8c966 not found: ID does not exist"
Sep 11 17:44:29.898666 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.898631    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"]
Sep 11 17:44:29.907926 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:29.907898    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/sub-test-model-db95fd-kserve-849776bf6c-cv2cw"]
Sep 11 17:44:30.055667 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:30.055634    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="434404c1-2a0e-4c82-ad92-05f53eb21111" path="/var/lib/kubelet/pods/434404c1-2a0e-4c82-ad92-05f53eb21111/volumes"
Sep 11 17:44:30.056056 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:30.056043    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee86935-b26d-4e03-8af2-ba3c8ac78055" path="/var/lib/kubelet/pods/7ee86935-b26d-4e03-8af2-ba3c8ac78055/volumes"
Sep 11 17:44:30.312064 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:30.312030    2560 scope.go:117] "RemoveContainer" containerID="368ad45e1aea29f3f6936a9801c96e7025cc5dc3870ff7b04dc11af425519e4d"
Sep 11 17:44:30.320657 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:30.320633    2560 scope.go:117] "RemoveContainer" containerID="68d80e65009815b7f38410428f22d1cda798063fe0f0833539d73335e078b31f"
Sep 11 17:44:30.329007 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:30.328983    2560 scope.go:117] "RemoveContainer" containerID="c4c1f613559d7e58fd4526e4b2a309b09191f489a76f1cc6f0b8295cfba96d79"
Sep 11 17:44:33.716637 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:33.716599    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:33.874808 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:33.874769    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:33.994590 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:33.994505    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"]
Sep 11 17:44:33.994782 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:33.994758    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b" podUID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerName="istio-proxy" containerID="cri-o://c6ca24933c2e9a5a018081d2b5e9e7fc2472e73d1f5b273e587f1ad8c5316349" gracePeriod=30
Sep 11 17:44:34.027831 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.027795    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:34.369739 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.369661    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms"]
Sep 11 17:44:34.370037 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.370026    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7102f469-a6f9-4502-82d4-dcab6acbfda3" containerName="istio-proxy"
Sep 11 17:44:34.370081 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.370039    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7102f469-a6f9-4502-82d4-dcab6acbfda3" containerName="istio-proxy"
Sep 11 17:44:34.370081 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.370048    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab3614e2-eb68-4368-abff-ce6ae5ac5347" containerName="revoke-keys"
Sep 11 17:44:34.370081 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.370053    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab3614e2-eb68-4368-abff-ce6ae5ac5347" containerName="revoke-keys"
Sep 11 17:44:34.370081 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.370065    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7102f469-a6f9-4502-82d4-dcab6acbfda3" containerName="istio-proxy"
Sep 11 17:44:34.370081 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.370070    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7102f469-a6f9-4502-82d4-dcab6acbfda3" containerName="istio-proxy"
Sep 11 17:44:34.370244 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.370084    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee86935-b26d-4e03-8af2-ba3c8ac78055" containerName="main"
Sep 11 17:44:34.370244 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.370089    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee86935-b26d-4e03-8af2-ba3c8ac78055" containerName="main"
Sep 11 17:44:34.370244 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.370100    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="434404c1-2a0e-4c82-ad92-05f53eb21111" containerName="main"
Sep 11 17:44:34.370244 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.370105    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="434404c1-2a0e-4c82-ad92-05f53eb21111" containerName="main"
Sep 11 17:44:34.370244 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.370177    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7102f469-a6f9-4502-82d4-dcab6acbfda3" containerName="istio-proxy"
Sep 11 17:44:34.370244 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.370187    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee86935-b26d-4e03-8af2-ba3c8ac78055" containerName="main"
Sep 11 17:44:34.370244 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.370195    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="434404c1-2a0e-4c82-ad92-05f53eb21111" containerName="main"
Sep 11 17:44:34.370244 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.370200    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab3614e2-eb68-4368-abff-ce6ae5ac5347" containerName="revoke-keys"
Sep 11 17:44:34.374954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.374938    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms"
Sep 11 17:44:34.379453 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.379432    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:44:34.379571 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.379485    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:44:34.386718 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.386693    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms"]
Sep 11 17:44:34.495869 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.495828    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ea1b18-d63e-4af3-ad3c-256436b29b84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms\" (UID: \"b9ea1b18-d63e-4af3-ad3c-256436b29b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms"
Sep 11 17:44:34.596780 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.596744    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ea1b18-d63e-4af3-ad3c-256436b29b84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms\" (UID: \"b9ea1b18-d63e-4af3-ad3c-256436b29b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms"
Sep 11 17:44:34.597353 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.597334    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ea1b18-d63e-4af3-ad3c-256436b29b84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms\" (UID: \"b9ea1b18-d63e-4af3-ad3c-256436b29b84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms"
Sep 11 17:44:34.685203 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.685167    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms"
Sep 11 17:44:34.709644 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.709575    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:34.990413 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:34.990389    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms"]
Sep 11 17:44:34.993113 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:44:34.993082    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ea1b18_d63e_4af3_ad3c_256436b29b84.slice/crio-b0b30f9e11a187a579b89e3954b2d51abdca6d719fe4ea4ff05432fe35149073 WatchSource:0}: Error finding container b0b30f9e11a187a579b89e3954b2d51abdca6d719fe4ea4ff05432fe35149073: Status 404 returned error can't find the container with id b0b30f9e11a187a579b89e3954b2d51abdca6d719fe4ea4ff05432fe35149073
Sep 11 17:44:35.848684 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:35.848647    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:35.903066 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:35.903030    2560 generic.go:356] "Generic (PLEG): container finished" podID="b9ea1b18-d63e-4af3-ad3c-256436b29b84" containerID="33eb4d9c3ffaf91e2e23fca6f83cdfb73306f226447960303368d170a1aeb8fe" exitCode=0
Sep 11 17:44:35.903235 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:35.903118    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms" event={"ID":"b9ea1b18-d63e-4af3-ad3c-256436b29b84","Type":"ContainerDied","Data":"33eb4d9c3ffaf91e2e23fca6f83cdfb73306f226447960303368d170a1aeb8fe"}
Sep 11 17:44:35.903235 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:35.903173    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms" event={"ID":"b9ea1b18-d63e-4af3-ad3c-256436b29b84","Type":"ContainerStarted","Data":"b0b30f9e11a187a579b89e3954b2d51abdca6d719fe4ea4ff05432fe35149073"}
Sep 11 17:44:36.984197 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:36.984141    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:37.044177 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:37.044150    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms"
Sep 11 17:44:37.124173 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:37.124129    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ea1b18-d63e-4af3-ad3c-256436b29b84-maas-api-service-ca\") pod \"b9ea1b18-d63e-4af3-ad3c-256436b29b84\" (UID: \"b9ea1b18-d63e-4af3-ad3c-256436b29b84\") "
Sep 11 17:44:37.124595 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:37.124570    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ea1b18-d63e-4af3-ad3c-256436b29b84-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b9ea1b18-d63e-4af3-ad3c-256436b29b84" (UID: "b9ea1b18-d63e-4af3-ad3c-256436b29b84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:37.124732 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:37.124712    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9ea1b18-d63e-4af3-ad3c-256436b29b84-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:37.914954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:37.914922    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms"
Sep 11 17:44:37.915135 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:37.914955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms" event={"ID":"b9ea1b18-d63e-4af3-ad3c-256436b29b84","Type":"ContainerDied","Data":"b0b30f9e11a187a579b89e3954b2d51abdca6d719fe4ea4ff05432fe35149073"}
Sep 11 17:44:37.915135 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:37.915003    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0b30f9e11a187a579b89e3954b2d51abdca6d719fe4ea4ff05432fe35149073"
Sep 11 17:44:38.239020 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:38.238933    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:38.327337 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:38.327296    2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 17:44:38.327518 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:38.327362    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b" podUID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 17:44:39.250550 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.250522    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:44:39.347524 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.347414    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616deae5-3a6f-46c5-8c5f-63ab3474767f-istiod-ca-cert\") pod \"616deae5-3a6f-46c5-8c5f-63ab3474767f\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") "
Sep 11 17:44:39.347524 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.347487    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqcx5\" (UniqueName: \"kubernetes.io/projected/616deae5-3a6f-46c5-8c5f-63ab3474767f-kube-api-access-pqcx5\") pod \"616deae5-3a6f-46c5-8c5f-63ab3474767f\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") "
Sep 11 17:44:39.347524 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.347525    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-workload-socket\") pod \"616deae5-3a6f-46c5-8c5f-63ab3474767f\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") "
Sep 11 17:44:39.347760 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.347733    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-workload-certs\") pod \"616deae5-3a6f-46c5-8c5f-63ab3474767f\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") "
Sep 11 17:44:39.347805 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.347782    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-envoy\") pod \"616deae5-3a6f-46c5-8c5f-63ab3474767f\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") "
Sep 11 17:44:39.347847 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.347836    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-credential-socket\") pod \"616deae5-3a6f-46c5-8c5f-63ab3474767f\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") "
Sep 11 17:44:39.347905 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.347842    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "616deae5-3a6f-46c5-8c5f-63ab3474767f" (UID: "616deae5-3a6f-46c5-8c5f-63ab3474767f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:39.347905 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.347887    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-data\") pod \"616deae5-3a6f-46c5-8c5f-63ab3474767f\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") "
Sep 11 17:44:39.347989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.347928    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-token\") pod \"616deae5-3a6f-46c5-8c5f-63ab3474767f\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") "
Sep 11 17:44:39.347989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.347924    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616deae5-3a6f-46c5-8c5f-63ab3474767f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "616deae5-3a6f-46c5-8c5f-63ab3474767f" (UID: "616deae5-3a6f-46c5-8c5f-63ab3474767f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:44:39.347989 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.347955    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-podinfo\") pod \"616deae5-3a6f-46c5-8c5f-63ab3474767f\" (UID: \"616deae5-3a6f-46c5-8c5f-63ab3474767f\") "
Sep 11 17:44:39.348148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.347991    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "616deae5-3a6f-46c5-8c5f-63ab3474767f" (UID: "616deae5-3a6f-46c5-8c5f-63ab3474767f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:39.348444 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.348183    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "616deae5-3a6f-46c5-8c5f-63ab3474767f" (UID: "616deae5-3a6f-46c5-8c5f-63ab3474767f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:39.348444 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.348266    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-data" (OuterVolumeSpecName: "istio-data") pod "616deae5-3a6f-46c5-8c5f-63ab3474767f" (UID: "616deae5-3a6f-46c5-8c5f-63ab3474767f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:39.348444 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.348363    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-data\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:39.348444 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.348380    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616deae5-3a6f-46c5-8c5f-63ab3474767f-istiod-ca-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:39.348444 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.348394    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-workload-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:39.348444 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.348409    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-workload-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:39.348444 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.348421    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-credential-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:39.350009 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.349983    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "616deae5-3a6f-46c5-8c5f-63ab3474767f" (UID: "616deae5-3a6f-46c5-8c5f-63ab3474767f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:44:39.350109 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.350059    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "616deae5-3a6f-46c5-8c5f-63ab3474767f" (UID: "616deae5-3a6f-46c5-8c5f-63ab3474767f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:44:39.350224 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.350207    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616deae5-3a6f-46c5-8c5f-63ab3474767f-kube-api-access-pqcx5" (OuterVolumeSpecName: "kube-api-access-pqcx5") pod "616deae5-3a6f-46c5-8c5f-63ab3474767f" (UID: "616deae5-3a6f-46c5-8c5f-63ab3474767f"). InnerVolumeSpecName "kube-api-access-pqcx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:39.350425 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.350381    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-token" (OuterVolumeSpecName: "istio-token") pod "616deae5-3a6f-46c5-8c5f-63ab3474767f" (UID: "616deae5-3a6f-46c5-8c5f-63ab3474767f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:44:39.449839 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.449794    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-token\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:39.449839 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.449829    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-podinfo\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:39.449839 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.449842    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqcx5\" (UniqueName: \"kubernetes.io/projected/616deae5-3a6f-46c5-8c5f-63ab3474767f-kube-api-access-pqcx5\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:39.450092 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.449855    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616deae5-3a6f-46c5-8c5f-63ab3474767f-istio-envoy\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:44:39.624789 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.624691    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:39.926036 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.925995    2560 generic.go:356] "Generic (PLEG): container finished" podID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerID="c6ca24933c2e9a5a018081d2b5e9e7fc2472e73d1f5b273e587f1ad8c5316349" exitCode=0
Sep 11 17:44:39.926258 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.926049    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"
Sep 11 17:44:39.926258 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.926087    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b" event={"ID":"616deae5-3a6f-46c5-8c5f-63ab3474767f","Type":"ContainerDied","Data":"c6ca24933c2e9a5a018081d2b5e9e7fc2472e73d1f5b273e587f1ad8c5316349"}
Sep 11 17:44:39.926258 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.926150    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b" event={"ID":"616deae5-3a6f-46c5-8c5f-63ab3474767f","Type":"ContainerDied","Data":"de9239484054fa611f1fd607f53d59c640294f430bcf83ac242fd2ff6ff6a627"}
Sep 11 17:44:39.926258 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.926184    2560 scope.go:117] "RemoveContainer" containerID="c6ca24933c2e9a5a018081d2b5e9e7fc2472e73d1f5b273e587f1ad8c5316349"
Sep 11 17:44:39.936545 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.936525    2560 scope.go:117] "RemoveContainer" containerID="b5def5d2cc566fef14185e06536a0239091db09c7e32bfe0f597bea7654cb0c3"
Sep 11 17:44:39.946860 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.946831    2560 scope.go:117] "RemoveContainer" containerID="c6ca24933c2e9a5a018081d2b5e9e7fc2472e73d1f5b273e587f1ad8c5316349"
Sep 11 17:44:39.947179 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:44:39.947160    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ca24933c2e9a5a018081d2b5e9e7fc2472e73d1f5b273e587f1ad8c5316349\": container with ID starting with c6ca24933c2e9a5a018081d2b5e9e7fc2472e73d1f5b273e587f1ad8c5316349 not found: ID does not exist" containerID="c6ca24933c2e9a5a018081d2b5e9e7fc2472e73d1f5b273e587f1ad8c5316349"
Sep 11 17:44:39.947228 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.947189    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ca24933c2e9a5a018081d2b5e9e7fc2472e73d1f5b273e587f1ad8c5316349"} err="failed to get container status \"c6ca24933c2e9a5a018081d2b5e9e7fc2472e73d1f5b273e587f1ad8c5316349\": rpc error: code = NotFound desc = could not find container \"c6ca24933c2e9a5a018081d2b5e9e7fc2472e73d1f5b273e587f1ad8c5316349\": container with ID starting with c6ca24933c2e9a5a018081d2b5e9e7fc2472e73d1f5b273e587f1ad8c5316349 not found: ID does not exist"
Sep 11 17:44:39.947228 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.947207    2560 scope.go:117] "RemoveContainer" containerID="b5def5d2cc566fef14185e06536a0239091db09c7e32bfe0f597bea7654cb0c3"
Sep 11 17:44:39.947493 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:44:39.947456    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5def5d2cc566fef14185e06536a0239091db09c7e32bfe0f597bea7654cb0c3\": container with ID starting with b5def5d2cc566fef14185e06536a0239091db09c7e32bfe0f597bea7654cb0c3 not found: ID does not exist" containerID="b5def5d2cc566fef14185e06536a0239091db09c7e32bfe0f597bea7654cb0c3"
Sep 11 17:44:39.947594 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.947576    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5def5d2cc566fef14185e06536a0239091db09c7e32bfe0f597bea7654cb0c3"} err="failed to get container status \"b5def5d2cc566fef14185e06536a0239091db09c7e32bfe0f597bea7654cb0c3\": rpc error: code = NotFound desc = could not find container \"b5def5d2cc566fef14185e06536a0239091db09c7e32bfe0f597bea7654cb0c3\": container with ID starting with b5def5d2cc566fef14185e06536a0239091db09c7e32bfe0f597bea7654cb0c3 not found: ID does not exist"
Sep 11 17:44:39.960094 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.960057    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"]
Sep 11 17:44:39.970142 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:39.970105    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-470521-openshift-default-cc7ccf6b8-vq64b"]
Sep 11 17:44:40.051262 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:40.051226    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="616deae5-3a6f-46c5-8c5f-63ab3474767f" path="/var/lib/kubelet/pods/616deae5-3a6f-46c5-8c5f-63ab3474767f/volumes"
Sep 11 17:44:45.366938 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:45.366898    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:45.484482 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:45.484432    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:45.658450 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:45.658409    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:46.004093 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:46.003996    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms"]
Sep 11 17:44:46.041632 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:46.041594    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b6b979-nsjms"]
Sep 11 17:44:46.050731 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:46.050703    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9ea1b18-d63e-4af3-ad3c-256436b29b84" path="/var/lib/kubelet/pods/b9ea1b18-d63e-4af3-ad3c-256436b29b84/volumes"
Sep 11 17:44:46.056385 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:46.056347    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:46.360260 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:46.360169    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:46.577500 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:46.577450    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:46.729296 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:46.729264    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:47.710675 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.710642    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:47.888192 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.888153    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"]
Sep 11 17:44:47.888568 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.888553    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerName="istio-proxy"
Sep 11 17:44:47.888620 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.888569    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerName="istio-proxy"
Sep 11 17:44:47.888620 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.888588    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ea1b18-d63e-4af3-ad3c-256436b29b84" containerName="revoke-keys"
Sep 11 17:44:47.888620 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.888597    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ea1b18-d63e-4af3-ad3c-256436b29b84" containerName="revoke-keys"
Sep 11 17:44:47.888715 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.888668    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerName="istio-proxy"
Sep 11 17:44:47.888715 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.888677    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerName="istio-proxy"
Sep 11 17:44:47.888715 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.888688    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9ea1b18-d63e-4af3-ad3c-256436b29b84" containerName="revoke-keys"
Sep 11 17:44:47.888715 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.888694    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7102f469-a6f9-4502-82d4-dcab6acbfda3" containerName="istio-proxy"
Sep 11 17:44:47.892646 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.892628    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:47.896715 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.896687    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4c36f4\"/\"kube-root-ca.crt\""
Sep 11 17:44:47.899346 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.899321    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4c36f4\"/\"default-dockercfg-s747w\""
Sep 11 17:44:47.899509 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.899326    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4c36f4\"/\"openshift-service-ca.crt\""
Sep 11 17:44:47.899509 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.899365    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4c36f4\"/\"rate-test-model-4c36f4-kserve-self-signed-certs\""
Sep 11 17:44:47.993690 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:47.993609    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"]
Sep 11 17:44:48.013928 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.013889    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:48.035964 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.035923    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-dshm\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.036128 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.035974    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-tmp-dir\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.036128 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.036021    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shrc2\" (UniqueName: \"kubernetes.io/projected/ab53fca9-2247-4418-aa6b-56b1644ffc2b-kube-api-access-shrc2\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.036128 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.036094    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab53fca9-2247-4418-aa6b-56b1644ffc2b-tls-certs\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.036239 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.036160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-home\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.036239 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.036198    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-model-cache\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.137574 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.137533    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab53fca9-2247-4418-aa6b-56b1644ffc2b-tls-certs\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.137792 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.137606    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-home\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.137792 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.137653    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-model-cache\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.137792 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.137703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-dshm\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.137792 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.137741    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-tmp-dir\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.137792 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.137782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shrc2\" (UniqueName: \"kubernetes.io/projected/ab53fca9-2247-4418-aa6b-56b1644ffc2b-kube-api-access-shrc2\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.138124 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.138093    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-home\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.138181 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.138105    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-model-cache\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.138229 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.138176    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-tmp-dir\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.139980 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.139959    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-dshm\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.140161 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.140142    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab53fca9-2247-4418-aa6b-56b1644ffc2b-tls-certs\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.150911 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.150881    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shrc2\" (UniqueName: \"kubernetes.io/projected/ab53fca9-2247-4418-aa6b-56b1644ffc2b-kube-api-access-shrc2\") pod \"rate-test-model-4c36f4-kserve-d89657867-vd8jx\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") " pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.202787 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.202745    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:48.494222 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.494194    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"]
Sep 11 17:44:48.496181 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:44:48.496150    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab53fca9_2247_4418_aa6b_56b1644ffc2b.slice/crio-c9878105b11bd8549afc824c4b05f9de71e924824051a7d6d7d665ab2e10e2cf WatchSource:0}: Error finding container c9878105b11bd8549afc824c4b05f9de71e924824051a7d6d7d665ab2e10e2cf: Status 404 returned error can't find the container with id c9878105b11bd8549afc824c4b05f9de71e924824051a7d6d7d665ab2e10e2cf
Sep 11 17:44:48.979484 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.979435    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx" event={"ID":"ab53fca9-2247-4418-aa6b-56b1644ffc2b","Type":"ContainerStarted","Data":"4384c5a50dd7ec5edc9c99efaea56acf6685bf254c13a237a6dd49662c0ed690"}
Sep 11 17:44:48.979484 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:48.979488    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx" event={"ID":"ab53fca9-2247-4418-aa6b-56b1644ffc2b","Type":"ContainerStarted","Data":"c9878105b11bd8549afc824c4b05f9de71e924824051a7d6d7d665ab2e10e2cf"}
Sep 11 17:44:49.032338 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:49.032283    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx" podStartSLOduration=2.032266496 podStartE2EDuration="2.032266496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:44:49.030450128 +0000 UTC m=+1459.462902905" watchObservedRunningTime="2026-09-11 17:44:49.032266496 +0000 UTC m=+1459.464719273"
Sep 11 17:44:49.051914 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:49.051882    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:53.324303 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:53.324273    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:53.440219 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:53.440175    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:53.619872 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:53.619776    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:44:58.203502 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:58.203430    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:58.204112 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:58.203514    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:58.216449 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:58.216418    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:44:59.043446 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:44:59.043416    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:45:00.141651 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:00.141611    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db95fd-29819145-9vxjg"]
Sep 11 17:45:00.142040 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:00.141997    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerName="istio-proxy"
Sep 11 17:45:00.142040 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:00.142010    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="616deae5-3a6f-46c5-8c5f-63ab3474767f" containerName="istio-proxy"
Sep 11 17:45:00.145290 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:00.145267    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4c36f4-29819145-85ph6"]
Sep 11 17:45:00.145418 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:00.145383    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db95fd-29819145-9vxjg"
Sep 11 17:45:00.148069 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:00.148042    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:45:00.148430 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:00.148413    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4c36f4-29819145-85ph6"
Sep 11 17:45:00.155133 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:00.155103    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db95fd-29819145-9vxjg"]
Sep 11 17:45:00.159356 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:00.159333    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4c36f4-29819145-85ph6"]
Sep 11 17:45:00.159575 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:00.159501    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db95fd-29819145-9vxjg"
Sep 11 17:45:00.170061 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:00.170039    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4c36f4-29819145-85ph6"
Sep 11 17:45:00.435796 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:00.435769    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db95fd-29819145-9vxjg"]
Sep 11 17:45:00.438371 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:45:00.438345    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e92682_44d3_45b5_b0cf_fee6e3255351.slice/crio-af9edecc04c3a4661304eee7d128c0e338afae3f1a049e406cc91a84a942ca51 WatchSource:0}: Error finding container af9edecc04c3a4661304eee7d128c0e338afae3f1a049e406cc91a84a942ca51: Status 404 returned error can't find the container with id af9edecc04c3a4661304eee7d128c0e338afae3f1a049e406cc91a84a942ca51
Sep 11 17:45:00.455579 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:00.455536    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4c36f4-29819145-85ph6"]
Sep 11 17:45:00.458168 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:45:00.458143    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2db8bb_6fa7_428e_9edd_6e132af5ff50.slice/crio-3aa3d9adb813cbbcd58ef1eef823f7558e005f6153ad1a3e8ef54d939e365bb8 WatchSource:0}: Error finding container 3aa3d9adb813cbbcd58ef1eef823f7558e005f6153ad1a3e8ef54d939e365bb8: Status 404 returned error can't find the container with id 3aa3d9adb813cbbcd58ef1eef823f7558e005f6153ad1a3e8ef54d939e365bb8
Sep 11 17:45:01.044462 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:01.044358    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4c36f4-29819145-85ph6" event={"ID":"1c2db8bb-6fa7-428e-9edd-6e132af5ff50","Type":"ContainerStarted","Data":"a94d469e5e02423e62a109df4f509da2f78523e63ed4bb2b20b91ee5483ac7e0"}
Sep 11 17:45:01.044462 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:01.044457    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4c36f4-29819145-85ph6" event={"ID":"1c2db8bb-6fa7-428e-9edd-6e132af5ff50","Type":"ContainerStarted","Data":"3aa3d9adb813cbbcd58ef1eef823f7558e005f6153ad1a3e8ef54d939e365bb8"}
Sep 11 17:45:01.046017 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:01.045988    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db95fd-29819145-9vxjg" event={"ID":"b0e92682-44d3-45b5-b0cf-fee6e3255351","Type":"ContainerStarted","Data":"040240859fe35a9ccc36b71144ffecf971c0eb774d2bc4b61d7179077c3a34b2"}
Sep 11 17:45:01.046160 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:01.046034    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db95fd-29819145-9vxjg" event={"ID":"b0e92682-44d3-45b5-b0cf-fee6e3255351","Type":"ContainerStarted","Data":"af9edecc04c3a4661304eee7d128c0e338afae3f1a049e406cc91a84a942ca51"}
Sep 11 17:45:01.062992 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:01.062933    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4c36f4-29819145-85ph6" podStartSLOduration=1.062914607 podStartE2EDuration="1.062914607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:45:01.061533047 +0000 UTC m=+1471.493985825" watchObservedRunningTime="2026-09-11 17:45:01.062914607 +0000 UTC m=+1471.495367384"
Sep 11 17:45:01.078656 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:01.078604    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db95fd-29819145-9vxjg" podStartSLOduration=1.078587216 podStartE2EDuration="1.078587216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:45:01.078302822 +0000 UTC m=+1471.510755600" watchObservedRunningTime="2026-09-11 17:45:01.078587216 +0000 UTC m=+1471.511039992"
Sep 11 17:45:02.052422 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:02.052390    2560 generic.go:356] "Generic (PLEG): container finished" podID="1c2db8bb-6fa7-428e-9edd-6e132af5ff50" containerID="a94d469e5e02423e62a109df4f509da2f78523e63ed4bb2b20b91ee5483ac7e0" exitCode=0
Sep 11 17:45:02.052873 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:02.052494    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4c36f4-29819145-85ph6" event={"ID":"1c2db8bb-6fa7-428e-9edd-6e132af5ff50","Type":"ContainerDied","Data":"a94d469e5e02423e62a109df4f509da2f78523e63ed4bb2b20b91ee5483ac7e0"}
Sep 11 17:45:02.053913 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:02.053893    2560 generic.go:356] "Generic (PLEG): container finished" podID="b0e92682-44d3-45b5-b0cf-fee6e3255351" containerID="040240859fe35a9ccc36b71144ffecf971c0eb774d2bc4b61d7179077c3a34b2" exitCode=0
Sep 11 17:45:02.054029 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:02.053973    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db95fd-29819145-9vxjg" event={"ID":"b0e92682-44d3-45b5-b0cf-fee6e3255351","Type":"ContainerDied","Data":"040240859fe35a9ccc36b71144ffecf971c0eb774d2bc4b61d7179077c3a34b2"}
Sep 11 17:45:03.218019 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:03.217990    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4c36f4-29819145-85ph6"
Sep 11 17:45:03.220974 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:03.220955    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db95fd-29819145-9vxjg"
Sep 11 17:45:04.065408 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:04.065384    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4c36f4-29819145-85ph6"
Sep 11 17:45:04.065559 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:04.065403    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4c36f4-29819145-85ph6" event={"ID":"1c2db8bb-6fa7-428e-9edd-6e132af5ff50","Type":"ContainerDied","Data":"3aa3d9adb813cbbcd58ef1eef823f7558e005f6153ad1a3e8ef54d939e365bb8"}
Sep 11 17:45:04.065559 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:04.065452    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3aa3d9adb813cbbcd58ef1eef823f7558e005f6153ad1a3e8ef54d939e365bb8"
Sep 11 17:45:04.066758 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:04.066734    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db95fd-29819145-9vxjg" event={"ID":"b0e92682-44d3-45b5-b0cf-fee6e3255351","Type":"ContainerDied","Data":"af9edecc04c3a4661304eee7d128c0e338afae3f1a049e406cc91a84a942ca51"}
Sep 11 17:45:04.066758 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:04.066749    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db95fd-29819145-9vxjg"
Sep 11 17:45:04.066926 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:04.066770    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af9edecc04c3a4661304eee7d128c0e338afae3f1a049e406cc91a84a942ca51"
Sep 11 17:45:07.154582 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.154547    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"]
Sep 11 17:45:07.155139 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.155118    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0e92682-44d3-45b5-b0cf-fee6e3255351" containerName="cleanup"
Sep 11 17:45:07.155226 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.155143    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0e92682-44d3-45b5-b0cf-fee6e3255351" containerName="cleanup"
Sep 11 17:45:07.155226 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.155166    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2db8bb-6fa7-428e-9edd-6e132af5ff50" containerName="cleanup"
Sep 11 17:45:07.155226 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.155174    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2db8bb-6fa7-428e-9edd-6e132af5ff50" containerName="cleanup"
Sep 11 17:45:07.155379 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.155288    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0e92682-44d3-45b5-b0cf-fee6e3255351" containerName="cleanup"
Sep 11 17:45:07.155379 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.155301    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2db8bb-6fa7-428e-9edd-6e132af5ff50" containerName="cleanup"
Sep 11 17:45:07.158260 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.158240    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.167012 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.166981    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-db95fd\"/\"kube-root-ca.crt\""
Sep 11 17:45:07.168095 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.168074    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-db95fd\"/\"rate-test-model-db95fd-kserve-self-signed-certs\""
Sep 11 17:45:07.168095 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.168087    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-db95fd\"/\"default-dockercfg-fhfrq\""
Sep 11 17:45:07.168222 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.168180    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-db95fd\"/\"openshift-service-ca.crt\""
Sep 11 17:45:07.176447 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.176419    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"]
Sep 11 17:45:07.222039 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.221998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-home\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.222191 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.222113    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43064d6e-8c53-4d67-a53d-abb50c1e1968-tls-certs\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.222191 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.222140    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtl7k\" (UniqueName: \"kubernetes.io/projected/43064d6e-8c53-4d67-a53d-abb50c1e1968-kube-api-access-vtl7k\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.222301 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.222231    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-model-cache\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.222301 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.222273    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-dshm\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.222389 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.222317    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-tmp-dir\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.275605 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.275561    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:07.323255 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.323206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-model-cache\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.323514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.323271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-dshm\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.323514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.323310    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-tmp-dir\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.323514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.323342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-home\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.323514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.323398    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43064d6e-8c53-4d67-a53d-abb50c1e1968-tls-certs\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.323514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.323421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtl7k\" (UniqueName: \"kubernetes.io/projected/43064d6e-8c53-4d67-a53d-abb50c1e1968-kube-api-access-vtl7k\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.323800 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.323712    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-model-cache\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.323800 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.323774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-tmp-dir\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.323915 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.323895    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-home\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.325695 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.325678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-dshm\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.325812 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.325795    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43064d6e-8c53-4d67-a53d-abb50c1e1968-tls-certs\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.332459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.332437    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtl7k\" (UniqueName: \"kubernetes.io/projected/43064d6e-8c53-4d67-a53d-abb50c1e1968-kube-api-access-vtl7k\") pod \"rate-test-model-db95fd-kserve-bdc84db99-h646c\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") " pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.467378 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.467279    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:07.754028 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:07.754000    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"]
Sep 11 17:45:07.756489 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:45:07.756449    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43064d6e_8c53_4d67_a53d_abb50c1e1968.slice/crio-d8ed93e801e3c3e6cf7a69f871311409ca8b2ed5afd3742187c5d7f6ebc553ab WatchSource:0}: Error finding container d8ed93e801e3c3e6cf7a69f871311409ca8b2ed5afd3742187c5d7f6ebc553ab: Status 404 returned error can't find the container with id d8ed93e801e3c3e6cf7a69f871311409ca8b2ed5afd3742187c5d7f6ebc553ab
Sep 11 17:45:08.091175 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:08.091075    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c" event={"ID":"43064d6e-8c53-4d67-a53d-abb50c1e1968","Type":"ContainerStarted","Data":"c2cdbb8de8930001fcd850412c2122e12790350d4c6644e821100f8ed4fbde15"}
Sep 11 17:45:08.091175 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:08.091121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c" event={"ID":"43064d6e-8c53-4d67-a53d-abb50c1e1968","Type":"ContainerStarted","Data":"d8ed93e801e3c3e6cf7a69f871311409ca8b2ed5afd3742187c5d7f6ebc553ab"}
Sep 11 17:45:08.112541 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:08.112454    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c" podStartSLOduration=1.112435884 podStartE2EDuration="1.112435884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:45:08.1103473 +0000 UTC m=+1478.542800080" watchObservedRunningTime="2026-09-11 17:45:08.112435884 +0000 UTC m=+1478.544888662"
Sep 11 17:45:17.467819 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:17.467774    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:17.468334 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:17.467830    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:17.480566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:17.480533    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:18.152755 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:18.152721    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:30.100010 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:30.099982    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:45:30.103736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:30.103712    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:45:42.877972 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:42.877930    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"]
Sep 11 17:45:43.940515 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:43.940455    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"]
Sep 11 17:45:44.071036 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:44.070990    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:44.144814 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:44.144779    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:44.473652 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:44.473617    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:44.851041 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:44.850947    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:45.332654 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:45.332619    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:48.067643 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:48.067602    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:48.169427 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:48.169392    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:48.277312 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:48.277271    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:49.049613 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:49.049580    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:50.061756 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:50.061725    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:50.160589 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:50.160549    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:50.267187 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:50.267156    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:50.547299 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:50.547250    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:50.647021 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:50.646987    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:53.786984 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:53.786891    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:57.897425 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:57.897385    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx" podUID="ab53fca9-2247-4418-aa6b-56b1644ffc2b" containerName="main" containerID="cri-o://4384c5a50dd7ec5edc9c99efaea56acf6685bf254c13a237a6dd49662c0ed690" gracePeriod=45
Sep 11 17:45:58.123945 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.123892    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:45:58.158585 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.158508    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:45:58.242509 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.242454    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab53fca9-2247-4418-aa6b-56b1644ffc2b-tls-certs\") pod \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") "
Sep 11 17:45:58.242723 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.242532    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-tmp-dir\") pod \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") "
Sep 11 17:45:58.242723 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.242580    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-home\") pod \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") "
Sep 11 17:45:58.242723 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.242611    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shrc2\" (UniqueName: \"kubernetes.io/projected/ab53fca9-2247-4418-aa6b-56b1644ffc2b-kube-api-access-shrc2\") pod \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") "
Sep 11 17:45:58.242723 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.242644    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-dshm\") pod \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") "
Sep 11 17:45:58.242723 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.242679    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-model-cache\") pod \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\" (UID: \"ab53fca9-2247-4418-aa6b-56b1644ffc2b\") "
Sep 11 17:45:58.243004 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.242859    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-home" (OuterVolumeSpecName: "home") pod "ab53fca9-2247-4418-aa6b-56b1644ffc2b" (UID: "ab53fca9-2247-4418-aa6b-56b1644ffc2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:58.243004 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.242876    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab53fca9-2247-4418-aa6b-56b1644ffc2b" (UID: "ab53fca9-2247-4418-aa6b-56b1644ffc2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:58.243105 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.243016    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab53fca9-2247-4418-aa6b-56b1644ffc2b" (UID: "ab53fca9-2247-4418-aa6b-56b1644ffc2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:58.243313 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.243291    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:45:58.243447 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.243400    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:45:58.243447 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.243415    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:45:58.244910 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.244867    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-dshm" (OuterVolumeSpecName: "dshm") pod "ab53fca9-2247-4418-aa6b-56b1644ffc2b" (UID: "ab53fca9-2247-4418-aa6b-56b1644ffc2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:58.244910 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.244893    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab53fca9-2247-4418-aa6b-56b1644ffc2b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab53fca9-2247-4418-aa6b-56b1644ffc2b" (UID: "ab53fca9-2247-4418-aa6b-56b1644ffc2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:45:58.245102 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.245079    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab53fca9-2247-4418-aa6b-56b1644ffc2b-kube-api-access-shrc2" (OuterVolumeSpecName: "kube-api-access-shrc2") pod "ab53fca9-2247-4418-aa6b-56b1644ffc2b" (UID: "ab53fca9-2247-4418-aa6b-56b1644ffc2b"). InnerVolumeSpecName "kube-api-access-shrc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:58.344093 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.344052    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shrc2\" (UniqueName: \"kubernetes.io/projected/ab53fca9-2247-4418-aa6b-56b1644ffc2b-kube-api-access-shrc2\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:45:58.344093 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.344084    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab53fca9-2247-4418-aa6b-56b1644ffc2b-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:45:58.344093 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.344094    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab53fca9-2247-4418-aa6b-56b1644ffc2b-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:45:58.365118 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.365078    2560 generic.go:356] "Generic (PLEG): container finished" podID="ab53fca9-2247-4418-aa6b-56b1644ffc2b" containerID="4384c5a50dd7ec5edc9c99efaea56acf6685bf254c13a237a6dd49662c0ed690" exitCode=0
Sep 11 17:45:58.365118 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.365119    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"
Sep 11 17:45:58.365353 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.365162    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx" event={"ID":"ab53fca9-2247-4418-aa6b-56b1644ffc2b","Type":"ContainerDied","Data":"4384c5a50dd7ec5edc9c99efaea56acf6685bf254c13a237a6dd49662c0ed690"}
Sep 11 17:45:58.365353 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.365203    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx" event={"ID":"ab53fca9-2247-4418-aa6b-56b1644ffc2b","Type":"ContainerDied","Data":"c9878105b11bd8549afc824c4b05f9de71e924824051a7d6d7d665ab2e10e2cf"}
Sep 11 17:45:58.365353 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.365225    2560 scope.go:117] "RemoveContainer" containerID="4384c5a50dd7ec5edc9c99efaea56acf6685bf254c13a237a6dd49662c0ed690"
Sep 11 17:45:58.375345 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.375321    2560 scope.go:117] "RemoveContainer" containerID="4384c5a50dd7ec5edc9c99efaea56acf6685bf254c13a237a6dd49662c0ed690"
Sep 11 17:45:58.375713 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:45:58.375693    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4384c5a50dd7ec5edc9c99efaea56acf6685bf254c13a237a6dd49662c0ed690\": container with ID starting with 4384c5a50dd7ec5edc9c99efaea56acf6685bf254c13a237a6dd49662c0ed690 not found: ID does not exist" containerID="4384c5a50dd7ec5edc9c99efaea56acf6685bf254c13a237a6dd49662c0ed690"
Sep 11 17:45:58.375804 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.375721    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4384c5a50dd7ec5edc9c99efaea56acf6685bf254c13a237a6dd49662c0ed690"} err="failed to get container status \"4384c5a50dd7ec5edc9c99efaea56acf6685bf254c13a237a6dd49662c0ed690\": rpc error: code = NotFound desc = could not find container \"4384c5a50dd7ec5edc9c99efaea56acf6685bf254c13a237a6dd49662c0ed690\": container with ID starting with 4384c5a50dd7ec5edc9c99efaea56acf6685bf254c13a237a6dd49662c0ed690 not found: ID does not exist"
Sep 11 17:45:58.394069 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.394034    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"]
Sep 11 17:45:58.395940 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.395915    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4c36f4/rate-test-model-4c36f4-kserve-d89657867-vd8jx"]
Sep 11 17:45:58.960034 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:58.959988    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c" podUID="43064d6e-8c53-4d67-a53d-abb50c1e1968" containerName="main" containerID="cri-o://c2cdbb8de8930001fcd850412c2122e12790350d4c6644e821100f8ed4fbde15" gracePeriod=45
Sep 11 17:45:59.216862 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.216787    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:59.252535 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.252504    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtl7k\" (UniqueName: \"kubernetes.io/projected/43064d6e-8c53-4d67-a53d-abb50c1e1968-kube-api-access-vtl7k\") pod \"43064d6e-8c53-4d67-a53d-abb50c1e1968\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") "
Sep 11 17:45:59.252733 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.252546    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-home\") pod \"43064d6e-8c53-4d67-a53d-abb50c1e1968\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") "
Sep 11 17:45:59.252733 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.252624    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-model-cache\") pod \"43064d6e-8c53-4d67-a53d-abb50c1e1968\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") "
Sep 11 17:45:59.252733 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.252660    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-tmp-dir\") pod \"43064d6e-8c53-4d67-a53d-abb50c1e1968\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") "
Sep 11 17:45:59.252733 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.252705    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43064d6e-8c53-4d67-a53d-abb50c1e1968-tls-certs\") pod \"43064d6e-8c53-4d67-a53d-abb50c1e1968\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") "
Sep 11 17:45:59.252971 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.252753    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-dshm\") pod \"43064d6e-8c53-4d67-a53d-abb50c1e1968\" (UID: \"43064d6e-8c53-4d67-a53d-abb50c1e1968\") "
Sep 11 17:45:59.252971 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.252873    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-home" (OuterVolumeSpecName: "home") pod "43064d6e-8c53-4d67-a53d-abb50c1e1968" (UID: "43064d6e-8c53-4d67-a53d-abb50c1e1968"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:59.252971 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.252915    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-model-cache" (OuterVolumeSpecName: "model-cache") pod "43064d6e-8c53-4d67-a53d-abb50c1e1968" (UID: "43064d6e-8c53-4d67-a53d-abb50c1e1968"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:59.252971 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.252929    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43064d6e-8c53-4d67-a53d-abb50c1e1968" (UID: "43064d6e-8c53-4d67-a53d-abb50c1e1968"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:59.253209 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.253144    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:45:59.253209 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.253173    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:45:59.253209 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.253187    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:45:59.254833 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.254809    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-dshm" (OuterVolumeSpecName: "dshm") pod "43064d6e-8c53-4d67-a53d-abb50c1e1968" (UID: "43064d6e-8c53-4d67-a53d-abb50c1e1968"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:45:59.254833 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.254823    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43064d6e-8c53-4d67-a53d-abb50c1e1968-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43064d6e-8c53-4d67-a53d-abb50c1e1968" (UID: "43064d6e-8c53-4d67-a53d-abb50c1e1968"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:45:59.255019 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.254815    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43064d6e-8c53-4d67-a53d-abb50c1e1968-kube-api-access-vtl7k" (OuterVolumeSpecName: "kube-api-access-vtl7k") pod "43064d6e-8c53-4d67-a53d-abb50c1e1968" (UID: "43064d6e-8c53-4d67-a53d-abb50c1e1968"). InnerVolumeSpecName "kube-api-access-vtl7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:45:59.354541 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.354506    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43064d6e-8c53-4d67-a53d-abb50c1e1968-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:45:59.354541 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.354534    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43064d6e-8c53-4d67-a53d-abb50c1e1968-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:45:59.354541 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.354543    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtl7k\" (UniqueName: \"kubernetes.io/projected/43064d6e-8c53-4d67-a53d-abb50c1e1968-kube-api-access-vtl7k\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:45:59.373270 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.373238    2560 generic.go:356] "Generic (PLEG): container finished" podID="43064d6e-8c53-4d67-a53d-abb50c1e1968" containerID="c2cdbb8de8930001fcd850412c2122e12790350d4c6644e821100f8ed4fbde15" exitCode=0
Sep 11 17:45:59.373270 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.373274    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"
Sep 11 17:45:59.373514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.373331    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c" event={"ID":"43064d6e-8c53-4d67-a53d-abb50c1e1968","Type":"ContainerDied","Data":"c2cdbb8de8930001fcd850412c2122e12790350d4c6644e821100f8ed4fbde15"}
Sep 11 17:45:59.373514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.373373    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c" event={"ID":"43064d6e-8c53-4d67-a53d-abb50c1e1968","Type":"ContainerDied","Data":"d8ed93e801e3c3e6cf7a69f871311409ca8b2ed5afd3742187c5d7f6ebc553ab"}
Sep 11 17:45:59.373514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.373390    2560 scope.go:117] "RemoveContainer" containerID="c2cdbb8de8930001fcd850412c2122e12790350d4c6644e821100f8ed4fbde15"
Sep 11 17:45:59.383279 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.383254    2560 scope.go:117] "RemoveContainer" containerID="c2cdbb8de8930001fcd850412c2122e12790350d4c6644e821100f8ed4fbde15"
Sep 11 17:45:59.383622 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:45:59.383602    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2cdbb8de8930001fcd850412c2122e12790350d4c6644e821100f8ed4fbde15\": container with ID starting with c2cdbb8de8930001fcd850412c2122e12790350d4c6644e821100f8ed4fbde15 not found: ID does not exist" containerID="c2cdbb8de8930001fcd850412c2122e12790350d4c6644e821100f8ed4fbde15"
Sep 11 17:45:59.383709 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.383630    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2cdbb8de8930001fcd850412c2122e12790350d4c6644e821100f8ed4fbde15"} err="failed to get container status \"c2cdbb8de8930001fcd850412c2122e12790350d4c6644e821100f8ed4fbde15\": rpc error: code = NotFound desc = could not find container \"c2cdbb8de8930001fcd850412c2122e12790350d4c6644e821100f8ed4fbde15\": container with ID starting with c2cdbb8de8930001fcd850412c2122e12790350d4c6644e821100f8ed4fbde15 not found: ID does not exist"
Sep 11 17:45:59.395731 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.395701    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"]
Sep 11 17:45:59.410973 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:45:59.410945    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-db95fd/rate-test-model-db95fd-kserve-bdc84db99-h646c"]
Sep 11 17:46:00.051684 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:00.051651    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43064d6e-8c53-4d67-a53d-abb50c1e1968" path="/var/lib/kubelet/pods/43064d6e-8c53-4d67-a53d-abb50c1e1968/volumes"
Sep 11 17:46:00.052099 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:00.052085    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab53fca9-2247-4418-aa6b-56b1644ffc2b" path="/var/lib/kubelet/pods/ab53fca9-2247-4418-aa6b-56b1644ffc2b/volumes"
Sep 11 17:46:01.473888 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:01.473854    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:12.298045 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:12.297983    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:13.128562 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:13.128521    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:13.228751 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:13.228717    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:13.907174 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:13.907140    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:14.007104 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.007072    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:14.225877 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.225792    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"]
Sep 11 17:46:14.226200 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.226188    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43064d6e-8c53-4d67-a53d-abb50c1e1968" containerName="main"
Sep 11 17:46:14.226253 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.226202    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="43064d6e-8c53-4d67-a53d-abb50c1e1968" containerName="main"
Sep 11 17:46:14.226253 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.226221    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab53fca9-2247-4418-aa6b-56b1644ffc2b" containerName="main"
Sep 11 17:46:14.226253 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.226226    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab53fca9-2247-4418-aa6b-56b1644ffc2b" containerName="main"
Sep 11 17:46:14.226361 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.226279    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab53fca9-2247-4418-aa6b-56b1644ffc2b" containerName="main"
Sep 11 17:46:14.226361 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.226286    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="43064d6e-8c53-4d67-a53d-abb50c1e1968" containerName="main"
Sep 11 17:46:14.229414 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.229394    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.232363 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.232339    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1d492f-openshift-default-dockercfg-5h52r\""
Sep 11 17:46:14.245016 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.244989    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"]
Sep 11 17:46:14.281970 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.281932    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:14.316210 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.316175    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-token\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.316403 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.316228    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-workload-certs\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.316403 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.316268    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-podinfo\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.316403 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.316338    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-envoy\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.316403 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.316374    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jvfn\" (UniqueName: \"kubernetes.io/projected/2472e12a-972b-4e94-9892-1a1ce97b8494-kube-api-access-5jvfn\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.316403 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.316403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2472e12a-972b-4e94-9892-1a1ce97b8494-istiod-ca-cert\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.316638 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.316518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-credential-socket\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.316638 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.316549    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-workload-socket\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.316638 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.316610    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-data\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.417359 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.417324    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-token\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.417604 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.417382    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-workload-certs\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.417604 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.417419    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-podinfo\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.417604 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.417503    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-envoy\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.417604 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.417530    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jvfn\" (UniqueName: \"kubernetes.io/projected/2472e12a-972b-4e94-9892-1a1ce97b8494-kube-api-access-5jvfn\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.417604 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.417564    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2472e12a-972b-4e94-9892-1a1ce97b8494-istiod-ca-cert\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.417879 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.417625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-credential-socket\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.417879 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.417655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-workload-socket\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.417879 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.417726    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-data\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.418047 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.417899    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-workload-certs\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.418105 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.418061    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-data\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.418183 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.418157    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-credential-socket\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.418281 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.418252    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-workload-socket\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.418395 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.418368    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2472e12a-972b-4e94-9892-1a1ce97b8494-istiod-ca-cert\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.420300 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.420274    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-envoy\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.420566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.420545    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-podinfo\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.429294 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.429266    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jvfn\" (UniqueName: \"kubernetes.io/projected/2472e12a-972b-4e94-9892-1a1ce97b8494-kube-api-access-5jvfn\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.429609 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.429588    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-token\") pod \"e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") " pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.539519 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.539403    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:14.692833 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.692799    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:14.823564 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.823536    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"]
Sep 11 17:46:14.825294 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:46:14.825263    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2472e12a_972b_4e94_9892_1a1ce97b8494.slice/crio-fadc7504d25d3e6f8d3039ddb3052ae97883ab29ec3de9c2f736d14eda0b460f WatchSource:0}: Error finding container fadc7504d25d3e6f8d3039ddb3052ae97883ab29ec3de9c2f736d14eda0b460f: Status 404 returned error can't find the container with id fadc7504d25d3e6f8d3039ddb3052ae97883ab29ec3de9c2f736d14eda0b460f
Sep 11 17:46:14.827544 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.827505    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:46:14.827632 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.827594    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:46:14.827632 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.827627    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:46:14.985647 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:14.985613    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:15.466083 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:15.466047    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls" event={"ID":"2472e12a-972b-4e94-9892-1a1ce97b8494","Type":"ContainerStarted","Data":"f3d20ea077bb68a1cfa11cd024e66ddbd43f445b6677c6006602734a533e46d4"}
Sep 11 17:46:15.466258 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:15.466098    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls" event={"ID":"2472e12a-972b-4e94-9892-1a1ce97b8494","Type":"ContainerStarted","Data":"fadc7504d25d3e6f8d3039ddb3052ae97883ab29ec3de9c2f736d14eda0b460f"}
Sep 11 17:46:15.501317 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:15.501246    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls" podStartSLOduration=1.501228078 podStartE2EDuration="1.501228078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:15.500649658 +0000 UTC m=+1545.933102436" watchObservedRunningTime="2026-09-11 17:46:15.501228078 +0000 UTC m=+1545.933680856"
Sep 11 17:46:15.540189 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:15.540151    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:15.545557 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:15.545528    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:16.471688 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:16.471656    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:16.472689 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:16.472671    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:16.841423 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:16.841340    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:17.544412 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:17.544378    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:17.645430 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:17.645392    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:17.767828 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:17.767786    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:18.749898 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:18.749860    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:19.611002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:19.610956    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:21.026581 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:21.026544    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:21.160438 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:21.160406    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:21.258042 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:21.258008    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:30.388446 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:30.388412    2560 scope.go:117] "RemoveContainer" containerID="ecc1bcc1cbe1704db5cec96083019a69ed651ccfa59a02fa77a16deb9095137e"
Sep 11 17:46:35.577748 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:35.577668    2560 generic.go:356] "Generic (PLEG): container finished" podID="2472e12a-972b-4e94-9892-1a1ce97b8494" containerID="f3d20ea077bb68a1cfa11cd024e66ddbd43f445b6677c6006602734a533e46d4" exitCode=0
Sep 11 17:46:35.578184 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:35.577743    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls" event={"ID":"2472e12a-972b-4e94-9892-1a1ce97b8494","Type":"ContainerDied","Data":"f3d20ea077bb68a1cfa11cd024e66ddbd43f445b6677c6006602734a533e46d4"}
Sep 11 17:46:35.578249 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:35.578229    2560 scope.go:117] "RemoveContainer" containerID="f3d20ea077bb68a1cfa11cd024e66ddbd43f445b6677c6006602734a533e46d4"
Sep 11 17:46:35.579375 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:35.579345    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:46:35.579442 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:35.579419    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:46:35.579505 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:35.579455    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:46:36.585042 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:36.585010    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls" event={"ID":"2472e12a-972b-4e94-9892-1a1ce97b8494","Type":"ContainerStarted","Data":"111f084aca00793834d030125d542d7c905368fd53506874bfc63bfe7e1151d3"}
Sep 11 17:46:37.539827 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:37.539787    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:37.540108 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:37.540084    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 11 17:46:37.540166 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:37.540146    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls" podUID="2472e12a-972b-4e94-9892-1a1ce97b8494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 11 17:46:38.540048 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:38.540004    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 11 17:46:38.540458 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:38.540077    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls" podUID="2472e12a-972b-4e94-9892-1a1ce97b8494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 11 17:46:39.540587 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:39.540550    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 11 17:46:39.540988 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:39.540624    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls" podUID="2472e12a-972b-4e94-9892-1a1ce97b8494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 11 17:46:40.540606 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:40.540561    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 11 17:46:40.541109 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:40.540633    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls" podUID="2472e12a-972b-4e94-9892-1a1ce97b8494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 11 17:46:41.540862 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:41.540812    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Sep 11 17:46:41.541266 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:41.540888    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls" podUID="2472e12a-972b-4e94-9892-1a1ce97b8494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Sep 11 17:46:42.533684 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:42.533650    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:42.545377 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:42.545345    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:42.545804 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:42.545759    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:42.546427 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:42.546409    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:46:42.744572 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:42.744539    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:43.575806 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:43.575773    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:44.389551 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:44.389517    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:45.530714 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:45.530679    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:46.290421 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:46.290388    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:48.410166 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:48.410127    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:50.034635 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.034595    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"]
Sep 11 17:46:50.037307 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.037289    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.039984 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.039960    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d8a0d0\"/\"openshift-service-ca.crt\""
Sep 11 17:46:50.040182 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.040169    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d8a0d0\"/\"default-dockercfg-vqpsr\""
Sep 11 17:46:50.040556 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.040540    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d8a0d0\"/\"kube-root-ca.crt\""
Sep 11 17:46:50.040951 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.040934    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d8a0d0\"/\"ipp-route-d8a0d0-kserve-self-signed-certs\""
Sep 11 17:46:50.052765 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.052736    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"]
Sep 11 17:46:50.130972 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.130937    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:46:50.186236 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.186190    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-tmp-dir\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.186236 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.186246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-model-cache\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.186543 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.186332    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48be2809-6027-471f-a140-44842e1e4af1-tls-certs\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.186543 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.186387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw58l\" (UniqueName: \"kubernetes.io/projected/48be2809-6027-471f-a140-44842e1e4af1-kube-api-access-hw58l\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.186543 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.186428    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-home\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.186703 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.186588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-dshm\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.288121 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.288021    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48be2809-6027-471f-a140-44842e1e4af1-tls-certs\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.288121 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.288088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw58l\" (UniqueName: \"kubernetes.io/projected/48be2809-6027-471f-a140-44842e1e4af1-kube-api-access-hw58l\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.288346 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.288278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-home\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.288388 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.288354    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-dshm\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.288423 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.288391    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-tmp-dir\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.288468 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.288434    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-model-cache\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.288773 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.288733    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-home\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.288873 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.288744    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-tmp-dir\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.288873 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.288797    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-model-cache\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.290578 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.290562    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-dshm\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.290652 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.290637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48be2809-6027-471f-a140-44842e1e4af1-tls-certs\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.296703 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.296678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw58l\" (UniqueName: \"kubernetes.io/projected/48be2809-6027-471f-a140-44842e1e4af1-kube-api-access-hw58l\") pod \"ipp-route-d8a0d0-kserve-86499b649f-gp746\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") " pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.347900 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.347854    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:46:50.621448 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.621421    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"]
Sep 11 17:46:50.623499 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:46:50.623456    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48be2809_6027_471f_a140_44842e1e4af1.slice/crio-3cc08b6922ad5c0f2bfbc1a61fcf11051d5cddfe69099f68a89f5150d9581a64 WatchSource:0}: Error finding container 3cc08b6922ad5c0f2bfbc1a61fcf11051d5cddfe69099f68a89f5150d9581a64: Status 404 returned error can't find the container with id 3cc08b6922ad5c0f2bfbc1a61fcf11051d5cddfe69099f68a89f5150d9581a64
Sep 11 17:46:50.678406 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:50.678375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746" event={"ID":"48be2809-6027-471f-a140-44842e1e4af1","Type":"ContainerStarted","Data":"3cc08b6922ad5c0f2bfbc1a61fcf11051d5cddfe69099f68a89f5150d9581a64"}
Sep 11 17:46:51.684935 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:51.684896    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746" event={"ID":"48be2809-6027-471f-a140-44842e1e4af1","Type":"ContainerStarted","Data":"467d517ecc9b2c46f044b60722eb8d6ede36df579568d0ce5b63b0038e71a392"}
Sep 11 17:46:51.702422 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:46:51.702360    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746" podStartSLOduration=1.7023419990000002 podStartE2EDuration="1.702341999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:46:51.701386297 +0000 UTC m=+1582.133839073" watchObservedRunningTime="2026-09-11 17:46:51.702341999 +0000 UTC m=+1582.134794775"
Sep 11 17:47:00.348939 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:00.348901    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:47:00.349379 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:00.348960    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:47:00.361717 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:00.361686    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:47:00.751286 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:00.751259    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:47:16.239346 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:16.239306    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:16.310339 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:16.310302    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:29.094559 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.094429    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:29.388186 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.388152    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"]
Sep 11 17:47:29.392005 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.391988    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.400408 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.400387    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-877b01-openshift-default-dockercfg-q4mbj\""
Sep 11 17:47:29.417324 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.417295    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:29.428505 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.428415    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"]
Sep 11 17:47:29.490736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.490697    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-workload-socket\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.490736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.490742    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-istio-data\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.491002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.490758    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4f308a-48e3-467c-813d-ced044138633-istio-token\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.491002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.490847    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4f308a-48e3-467c-813d-ced044138633-istiod-ca-cert\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.491002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.490898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-istio-envoy\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.491002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.490922    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4f308a-48e3-467c-813d-ced044138633-istio-podinfo\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.491002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.490988    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvm4s\" (UniqueName: \"kubernetes.io/projected/6f4f308a-48e3-467c-813d-ced044138633-kube-api-access-hvm4s\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.491246 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.491063    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-workload-certs\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.491246 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.491115    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-credential-socket\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.570036 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.569995    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:29.592293 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.592260    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvm4s\" (UniqueName: \"kubernetes.io/projected/6f4f308a-48e3-467c-813d-ced044138633-kube-api-access-hvm4s\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.592459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.592308    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-workload-certs\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.592459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.592339    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-credential-socket\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.592459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.592381    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-workload-socket\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.592459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.592401    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-istio-data\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.592459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.592420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4f308a-48e3-467c-813d-ced044138633-istio-token\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.592459 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.592452    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4f308a-48e3-467c-813d-ced044138633-istiod-ca-cert\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.592817 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.592518    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-istio-envoy\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.592817 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.592541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4f308a-48e3-467c-813d-ced044138633-istio-podinfo\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.592926 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.592878    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-workload-certs\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.592926 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.592911    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-workload-socket\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.593010 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.592993    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-credential-socket\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.593067 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.593046    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-istio-data\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.593334 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.593307    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4f308a-48e3-467c-813d-ced044138633-istiod-ca-cert\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.594924 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.594901    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-istio-envoy\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.595160 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.595139    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4f308a-48e3-467c-813d-ced044138633-istio-podinfo\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.608162 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.608135    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4f308a-48e3-467c-813d-ced044138633-istio-token\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.614287 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.614264    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvm4s\" (UniqueName: \"kubernetes.io/projected/6f4f308a-48e3-467c-813d-ced044138633-kube-api-access-hvm4s\") pod \"e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") " pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.702848 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.702755    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:29.984076 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.984043    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"]
Sep 11 17:47:29.985442 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:47:29.985415    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4f308a_48e3_467c_813d_ced044138633.slice/crio-26cd03d247e4f2ba45129222ad615211cf2134d8a9815a6e6184844d2c0c614c WatchSource:0}: Error finding container 26cd03d247e4f2ba45129222ad615211cf2134d8a9815a6e6184844d2c0c614c: Status 404 returned error can't find the container with id 26cd03d247e4f2ba45129222ad615211cf2134d8a9815a6e6184844d2c0c614c
Sep 11 17:47:29.987823 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.987787    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:47:29.987909 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.987862    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:47:29.987909 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:29.987893    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:47:30.903555 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:30.903515    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5" event={"ID":"6f4f308a-48e3-467c-813d-ced044138633","Type":"ContainerStarted","Data":"76d4b4415c38059c85cdf47cfaa267ce44b134e6775d25e35a0d545432b1da58"}
Sep 11 17:47:30.903956 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:30.903626    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5" event={"ID":"6f4f308a-48e3-467c-813d-ced044138633","Type":"ContainerStarted","Data":"26cd03d247e4f2ba45129222ad615211cf2134d8a9815a6e6184844d2c0c614c"}
Sep 11 17:47:30.932630 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:30.932566    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5" podStartSLOduration=1.9325468749999999 podStartE2EDuration="1.932546875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:47:30.929194522 +0000 UTC m=+1621.361647299" watchObservedRunningTime="2026-09-11 17:47:30.932546875 +0000 UTC m=+1621.364999652"
Sep 11 17:47:31.703081 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:31.703041    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:31.708443 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:31.708417    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:31.909308 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:31.909281    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:31.910185 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:31.910164    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:32.216373 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:32.216336    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:32.858687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:32.858646    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:32.959552 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:32.959517    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:33.092440 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:33.092407    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:34.041391 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:34.041358    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:34.765044 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:34.764992    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:35.635864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:35.635829    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:36.567341 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:36.567304    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:36.660589 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:36.660555    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:37.095978 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:37.095919    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:37.192712 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:37.192680    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:37.276467 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:37.276413    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:54.041620 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:54.041588    2560 generic.go:356] "Generic (PLEG): container finished" podID="6f4f308a-48e3-467c-813d-ced044138633" containerID="76d4b4415c38059c85cdf47cfaa267ce44b134e6775d25e35a0d545432b1da58" exitCode=0
Sep 11 17:47:54.042141 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:54.041663    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5" event={"ID":"6f4f308a-48e3-467c-813d-ced044138633","Type":"ContainerDied","Data":"76d4b4415c38059c85cdf47cfaa267ce44b134e6775d25e35a0d545432b1da58"}
Sep 11 17:47:54.042260 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:54.042241    2560 scope.go:117] "RemoveContainer" containerID="76d4b4415c38059c85cdf47cfaa267ce44b134e6775d25e35a0d545432b1da58"
Sep 11 17:47:54.043602 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:54.043567    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:47:54.043707 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:54.043649    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:47:54.043747 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:54.043715    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:47:55.049120 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:55.049086    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5" event={"ID":"6f4f308a-48e3-467c-813d-ced044138633","Type":"ContainerStarted","Data":"706a8ef9cf33bc55fdbd8e744b3a212ab60d020551a2a1e7af1cac7159f87521"}
Sep 11 17:47:55.703742 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:55.703700    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:47:55.703954 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:55.703931    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 11 17:47:55.704006 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:55.703982    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5" podUID="6f4f308a-48e3-467c-813d-ced044138633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 11 17:47:56.703977 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:56.703934    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 11 17:47:56.704400 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:56.703995    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5" podUID="6f4f308a-48e3-467c-813d-ced044138633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 11 17:47:57.703378 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:57.703337    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 11 17:47:57.703612 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:57.703405    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5" podUID="6f4f308a-48e3-467c-813d-ced044138633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 11 17:47:58.703334 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:58.703297    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 11 17:47:58.703784 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:58.703359    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5" podUID="6f4f308a-48e3-467c-813d-ced044138633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 11 17:47:58.757806 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:58.757771    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:59.521648 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:59.521607    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:47:59.703924 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:59.703882    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 11 17:47:59.704420 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:47:59.703963    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5" podUID="6f4f308a-48e3-467c-813d-ced044138633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 11 17:48:00.181597 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:00.181561    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:00.729896 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:00.729858    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:48:00.730393 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:00.730350    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:48:00.731207 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:00.731181    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:48:00.866798 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:00.866763    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:01.565046 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:01.565012    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:02.375098 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:02.375066    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:02.985736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:02.985706    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:03.712455 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:03.712415    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6"]
Sep 11 17:48:03.715769 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:03.715744    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6"
Sep 11 17:48:03.718553 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:03.718530    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:48:03.718804 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:03.718787    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:48:03.726110 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:03.726080    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6"]
Sep 11 17:48:03.856712 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:03.856667    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05741498-37d5-4411-97fc-3fffa5438b52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6\" (UID: \"05741498-37d5-4411-97fc-3fffa5438b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6"
Sep 11 17:48:03.957544 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:03.957469    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05741498-37d5-4411-97fc-3fffa5438b52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6\" (UID: \"05741498-37d5-4411-97fc-3fffa5438b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6"
Sep 11 17:48:03.958149 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:03.958126    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05741498-37d5-4411-97fc-3fffa5438b52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6\" (UID: \"05741498-37d5-4411-97fc-3fffa5438b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6"
Sep 11 17:48:04.027382 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:04.027280    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6"
Sep 11 17:48:04.300925 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:04.300897    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6"]
Sep 11 17:48:04.302830 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:48:04.302799    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05741498_37d5_4411_97fc_3fffa5438b52.slice/crio-97d6a77c0939ac122f47ca6cfe7d1b7fb43010627a30bdbf16a4aff43a3d7bba WatchSource:0}: Error finding container 97d6a77c0939ac122f47ca6cfe7d1b7fb43010627a30bdbf16a4aff43a3d7bba: Status 404 returned error can't find the container with id 97d6a77c0939ac122f47ca6cfe7d1b7fb43010627a30bdbf16a4aff43a3d7bba
Sep 11 17:48:04.442582 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:04.442522    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:05.109046 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:05.109014    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6" event={"ID":"05741498-37d5-4411-97fc-3fffa5438b52","Type":"ContainerStarted","Data":"144ab59dc91e6f01b49507db60c655f0c364c5a8f091ea4e6d4f36ebbd57e2b4"}
Sep 11 17:48:05.109536 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:05.109070    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6" event={"ID":"05741498-37d5-4411-97fc-3fffa5438b52","Type":"ContainerStarted","Data":"97d6a77c0939ac122f47ca6cfe7d1b7fb43010627a30bdbf16a4aff43a3d7bba"}
Sep 11 17:48:05.125324 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:05.125272    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6" podStartSLOduration=2.125256375 podStartE2EDuration="2.125256375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:48:05.123603928 +0000 UTC m=+1655.556056706" watchObservedRunningTime="2026-09-11 17:48:05.125256375 +0000 UTC m=+1655.557709151"
Sep 11 17:48:05.220352 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:05.220309    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:05.909282 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:05.909240    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:06.116159 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:06.116125    2560 generic.go:356] "Generic (PLEG): container finished" podID="05741498-37d5-4411-97fc-3fffa5438b52" containerID="144ab59dc91e6f01b49507db60c655f0c364c5a8f091ea4e6d4f36ebbd57e2b4" exitCode=0
Sep 11 17:48:06.116668 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:06.116215    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6" event={"ID":"05741498-37d5-4411-97fc-3fffa5438b52","Type":"ContainerDied","Data":"144ab59dc91e6f01b49507db60c655f0c364c5a8f091ea4e6d4f36ebbd57e2b4"}
Sep 11 17:48:06.633271 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:06.633231    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:07.262068 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:07.262043    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6"
Sep 11 17:48:07.289945 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:07.289913    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05741498-37d5-4411-97fc-3fffa5438b52-maas-api-service-ca\") pod \"05741498-37d5-4411-97fc-3fffa5438b52\" (UID: \"05741498-37d5-4411-97fc-3fffa5438b52\") "
Sep 11 17:48:07.290279 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:07.290253    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05741498-37d5-4411-97fc-3fffa5438b52-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05741498-37d5-4411-97fc-3fffa5438b52" (UID: "05741498-37d5-4411-97fc-3fffa5438b52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:48:07.391149 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:07.391111    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05741498-37d5-4411-97fc-3fffa5438b52-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:48:07.446858 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:07.446824    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:08.129582 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:08.129546    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6" event={"ID":"05741498-37d5-4411-97fc-3fffa5438b52","Type":"ContainerDied","Data":"97d6a77c0939ac122f47ca6cfe7d1b7fb43010627a30bdbf16a4aff43a3d7bba"}
Sep 11 17:48:08.129763 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:08.129602    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97d6a77c0939ac122f47ca6cfe7d1b7fb43010627a30bdbf16a4aff43a3d7bba"
Sep 11 17:48:08.129763 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:08.129561    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6"
Sep 11 17:48:08.186322 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:08.186292    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:14.670840 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:14.670799    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:14.748794 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:14.748753    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:14.847946 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:14.847915    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:14.886133 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:14.886096    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6"]
Sep 11 17:48:14.889906 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:14.889877    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-877b01-kvcf6"]
Sep 11 17:48:14.933822 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:14.933737    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:15.288579 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:15.288464    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:15.384730 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:15.384693    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:16.058234 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:16.058199    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05741498-37d5-4411-97fc-3fffa5438b52" path="/var/lib/kubelet/pods/05741498-37d5-4411-97fc-3fffa5438b52/volumes"
Sep 11 17:48:16.329186 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:16.329096    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:17.111944 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:17.111906    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:18.230294 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:18.230248    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:19.371768 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:19.371721    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:21.472861 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:21.472830    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:21.572776 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:21.572747    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:21.729442 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:21.729351    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"]
Sep 11 17:48:21.729625 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:21.729590    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5" podUID="6f4f308a-48e3-467c-813d-ced044138633" containerName="istio-proxy" containerID="cri-o://706a8ef9cf33bc55fdbd8e744b3a212ab60d020551a2a1e7af1cac7159f87521" gracePeriod=30
Sep 11 17:48:21.757336 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:21.757302    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:25.721090 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:25.721058    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5"]
Sep 11 17:48:25.721576 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:25.721448    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05741498-37d5-4411-97fc-3fffa5438b52" containerName="revoke-keys"
Sep 11 17:48:25.721576 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:25.721459    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="05741498-37d5-4411-97fc-3fffa5438b52" containerName="revoke-keys"
Sep 11 17:48:25.721576 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:25.721539    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="05741498-37d5-4411-97fc-3fffa5438b52" containerName="revoke-keys"
Sep 11 17:48:25.724839 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:25.724819    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5"
Sep 11 17:48:25.731539 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:25.731115    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:48:25.731539 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:25.731117    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:48:25.736303 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:25.736280    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5"]
Sep 11 17:48:25.876564 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:25.876535    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:25.881798 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:25.881774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41b8708-23cf-409a-a8ed-ed8fba05c4bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5\" (UID: \"e41b8708-23cf-409a-a8ed-ed8fba05c4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5"
Sep 11 17:48:25.982940 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:25.982838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41b8708-23cf-409a-a8ed-ed8fba05c4bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5\" (UID: \"e41b8708-23cf-409a-a8ed-ed8fba05c4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5"
Sep 11 17:48:25.983522 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:25.983498    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41b8708-23cf-409a-a8ed-ed8fba05c4bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5\" (UID: \"e41b8708-23cf-409a-a8ed-ed8fba05c4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5"
Sep 11 17:48:26.034987 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:26.034954    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5"
Sep 11 17:48:26.331530 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:26.331408    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5"]
Sep 11 17:48:26.335963 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:48:26.335936    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41b8708_23cf_409a_a8ed_ed8fba05c4bd.slice/crio-0162ef3e8852f0aeebb0964eee5e4cbdda6913626aa2be097388815e19703d3a WatchSource:0}: Error finding container 0162ef3e8852f0aeebb0964eee5e4cbdda6913626aa2be097388815e19703d3a: Status 404 returned error can't find the container with id 0162ef3e8852f0aeebb0964eee5e4cbdda6913626aa2be097388815e19703d3a
Sep 11 17:48:26.940315 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:26.940276    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:26.980695 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:26.980667    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:48:27.094081 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094042    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-istio-envoy\") pod \"6f4f308a-48e3-467c-813d-ced044138633\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") "
Sep 11 17:48:27.094081 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094088    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvm4s\" (UniqueName: \"kubernetes.io/projected/6f4f308a-48e3-467c-813d-ced044138633-kube-api-access-hvm4s\") pod \"6f4f308a-48e3-467c-813d-ced044138633\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") "
Sep 11 17:48:27.094357 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094110    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-istio-data\") pod \"6f4f308a-48e3-467c-813d-ced044138633\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") "
Sep 11 17:48:27.094357 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094150    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4f308a-48e3-467c-813d-ced044138633-istiod-ca-cert\") pod \"6f4f308a-48e3-467c-813d-ced044138633\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") "
Sep 11 17:48:27.094357 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094175    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-credential-socket\") pod \"6f4f308a-48e3-467c-813d-ced044138633\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") "
Sep 11 17:48:27.094357 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094215    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4f308a-48e3-467c-813d-ced044138633-istio-podinfo\") pod \"6f4f308a-48e3-467c-813d-ced044138633\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") "
Sep 11 17:48:27.094357 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094257    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-workload-certs\") pod \"6f4f308a-48e3-467c-813d-ced044138633\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") "
Sep 11 17:48:27.094357 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094309    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4f308a-48e3-467c-813d-ced044138633-istio-token\") pod \"6f4f308a-48e3-467c-813d-ced044138633\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") "
Sep 11 17:48:27.094727 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094377    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-workload-socket\") pod \"6f4f308a-48e3-467c-813d-ced044138633\" (UID: \"6f4f308a-48e3-467c-813d-ced044138633\") "
Sep 11 17:48:27.094727 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094584    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f4f308a-48e3-467c-813d-ced044138633" (UID: "6f4f308a-48e3-467c-813d-ced044138633"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:48:27.094727 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094697    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f4f308a-48e3-467c-813d-ced044138633" (UID: "6f4f308a-48e3-467c-813d-ced044138633"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:48:27.094892 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094744    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4f308a-48e3-467c-813d-ced044138633-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f4f308a-48e3-467c-813d-ced044138633" (UID: "6f4f308a-48e3-467c-813d-ced044138633"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:48:27.094892 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094864    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4f308a-48e3-467c-813d-ced044138633-istiod-ca-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:48:27.094892 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094889    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-credential-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:48:27.095049 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094904    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-workload-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:48:27.095049 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094862    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f4f308a-48e3-467c-813d-ced044138633" (UID: "6f4f308a-48e3-467c-813d-ced044138633"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:48:27.095049 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.094909    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f4f308a-48e3-467c-813d-ced044138633" (UID: "6f4f308a-48e3-467c-813d-ced044138633"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:48:27.096525 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.096499    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4f308a-48e3-467c-813d-ced044138633-kube-api-access-hvm4s" (OuterVolumeSpecName: "kube-api-access-hvm4s") pod "6f4f308a-48e3-467c-813d-ced044138633" (UID: "6f4f308a-48e3-467c-813d-ced044138633"). InnerVolumeSpecName "kube-api-access-hvm4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:48:27.096628 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.096542    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4f308a-48e3-467c-813d-ced044138633-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f4f308a-48e3-467c-813d-ced044138633" (UID: "6f4f308a-48e3-467c-813d-ced044138633"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:48:27.096628 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.096547    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f4f308a-48e3-467c-813d-ced044138633-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f4f308a-48e3-467c-813d-ced044138633" (UID: "6f4f308a-48e3-467c-813d-ced044138633"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:48:27.096628 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.096537    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f4f308a-48e3-467c-813d-ced044138633" (UID: "6f4f308a-48e3-467c-813d-ced044138633"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:48:27.196039 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.196005    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4f308a-48e3-467c-813d-ced044138633-istio-token\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:48:27.196039 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.196033    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-workload-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:48:27.196039 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.196042    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-istio-envoy\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:48:27.196282 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.196053    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvm4s\" (UniqueName: \"kubernetes.io/projected/6f4f308a-48e3-467c-813d-ced044138633-kube-api-access-hvm4s\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:48:27.196282 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.196062    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4f308a-48e3-467c-813d-ced044138633-istio-data\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:48:27.196282 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.196070    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4f308a-48e3-467c-813d-ced044138633-istio-podinfo\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:48:27.242953 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.242911    2560 generic.go:356] "Generic (PLEG): container finished" podID="6f4f308a-48e3-467c-813d-ced044138633" containerID="706a8ef9cf33bc55fdbd8e744b3a212ab60d020551a2a1e7af1cac7159f87521" exitCode=0
Sep 11 17:48:27.243148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.242977    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"
Sep 11 17:48:27.243148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.242993    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5" event={"ID":"6f4f308a-48e3-467c-813d-ced044138633","Type":"ContainerDied","Data":"706a8ef9cf33bc55fdbd8e744b3a212ab60d020551a2a1e7af1cac7159f87521"}
Sep 11 17:48:27.243148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.243081    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5" event={"ID":"6f4f308a-48e3-467c-813d-ced044138633","Type":"ContainerDied","Data":"26cd03d247e4f2ba45129222ad615211cf2134d8a9815a6e6184844d2c0c614c"}
Sep 11 17:48:27.243148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.243104    2560 scope.go:117] "RemoveContainer" containerID="706a8ef9cf33bc55fdbd8e744b3a212ab60d020551a2a1e7af1cac7159f87521"
Sep 11 17:48:27.244995 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.244969    2560 generic.go:356] "Generic (PLEG): container finished" podID="e41b8708-23cf-409a-a8ed-ed8fba05c4bd" containerID="78c62faaa362ae77d626f7ccacf7b123d737e9f2bafc6f299de2bb668a387b1b" exitCode=0
Sep 11 17:48:27.245089 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.245003    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5" event={"ID":"e41b8708-23cf-409a-a8ed-ed8fba05c4bd","Type":"ContainerDied","Data":"78c62faaa362ae77d626f7ccacf7b123d737e9f2bafc6f299de2bb668a387b1b"}
Sep 11 17:48:27.245089 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.245055    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5" event={"ID":"e41b8708-23cf-409a-a8ed-ed8fba05c4bd","Type":"ContainerStarted","Data":"0162ef3e8852f0aeebb0964eee5e4cbdda6913626aa2be097388815e19703d3a"}
Sep 11 17:48:27.253387 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.253368    2560 scope.go:117] "RemoveContainer" containerID="76d4b4415c38059c85cdf47cfaa267ce44b134e6775d25e35a0d545432b1da58"
Sep 11 17:48:27.261984 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.261961    2560 scope.go:117] "RemoveContainer" containerID="706a8ef9cf33bc55fdbd8e744b3a212ab60d020551a2a1e7af1cac7159f87521"
Sep 11 17:48:27.262276 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:48:27.262254    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706a8ef9cf33bc55fdbd8e744b3a212ab60d020551a2a1e7af1cac7159f87521\": container with ID starting with 706a8ef9cf33bc55fdbd8e744b3a212ab60d020551a2a1e7af1cac7159f87521 not found: ID does not exist" containerID="706a8ef9cf33bc55fdbd8e744b3a212ab60d020551a2a1e7af1cac7159f87521"
Sep 11 17:48:27.262321 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.262287    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706a8ef9cf33bc55fdbd8e744b3a212ab60d020551a2a1e7af1cac7159f87521"} err="failed to get container status \"706a8ef9cf33bc55fdbd8e744b3a212ab60d020551a2a1e7af1cac7159f87521\": rpc error: code = NotFound desc = could not find container \"706a8ef9cf33bc55fdbd8e744b3a212ab60d020551a2a1e7af1cac7159f87521\": container with ID starting with 706a8ef9cf33bc55fdbd8e744b3a212ab60d020551a2a1e7af1cac7159f87521 not found: ID does not exist"
Sep 11 17:48:27.262321 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.262307    2560 scope.go:117] "RemoveContainer" containerID="76d4b4415c38059c85cdf47cfaa267ce44b134e6775d25e35a0d545432b1da58"
Sep 11 17:48:27.262566 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:48:27.262547    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d4b4415c38059c85cdf47cfaa267ce44b134e6775d25e35a0d545432b1da58\": container with ID starting with 76d4b4415c38059c85cdf47cfaa267ce44b134e6775d25e35a0d545432b1da58 not found: ID does not exist" containerID="76d4b4415c38059c85cdf47cfaa267ce44b134e6775d25e35a0d545432b1da58"
Sep 11 17:48:27.262610 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.262574    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d4b4415c38059c85cdf47cfaa267ce44b134e6775d25e35a0d545432b1da58"} err="failed to get container status \"76d4b4415c38059c85cdf47cfaa267ce44b134e6775d25e35a0d545432b1da58\": rpc error: code = NotFound desc = could not find container \"76d4b4415c38059c85cdf47cfaa267ce44b134e6775d25e35a0d545432b1da58\": container with ID starting with 76d4b4415c38059c85cdf47cfaa267ce44b134e6775d25e35a0d545432b1da58 not found: ID does not exist"
Sep 11 17:48:27.296333 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.296298    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"]
Sep 11 17:48:27.300327 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.300299    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-877b01-openshift-default-7b8b7978-5ftn5"]
Sep 11 17:48:27.926019 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:27.925983    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:28.051584 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:28.051549    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f4f308a-48e3-467c-813d-ced044138633" path="/var/lib/kubelet/pods/6f4f308a-48e3-467c-813d-ced044138633/volumes"
Sep 11 17:48:28.388822 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:28.388795    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5"
Sep 11 17:48:28.509302 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:28.509204    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41b8708-23cf-409a-a8ed-ed8fba05c4bd-maas-api-service-ca\") pod \"e41b8708-23cf-409a-a8ed-ed8fba05c4bd\" (UID: \"e41b8708-23cf-409a-a8ed-ed8fba05c4bd\") "
Sep 11 17:48:28.509621 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:28.509599    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41b8708-23cf-409a-a8ed-ed8fba05c4bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e41b8708-23cf-409a-a8ed-ed8fba05c4bd" (UID: "e41b8708-23cf-409a-a8ed-ed8fba05c4bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:48:28.509709 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:28.509694    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41b8708-23cf-409a-a8ed-ed8fba05c4bd-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:48:28.601967 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:28.601919    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:29.258176 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:29.258145    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5"
Sep 11 17:48:29.258580 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:29.258191    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5" event={"ID":"e41b8708-23cf-409a-a8ed-ed8fba05c4bd","Type":"ContainerDied","Data":"0162ef3e8852f0aeebb0964eee5e4cbdda6913626aa2be097388815e19703d3a"}
Sep 11 17:48:29.258580 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:29.258236    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0162ef3e8852f0aeebb0964eee5e4cbdda6913626aa2be097388815e19703d3a"
Sep 11 17:48:29.300172 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:29.300137    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:30.049532 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:30.049468    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:41.261422 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:41.261379    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:41.386814 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:41.386757    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:41.513755 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:41.513675    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:41.609933 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:41.609892    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5"]
Sep 11 17:48:41.611309 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:41.611283    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:41.614563 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:41.614539    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d8a0d0-rkbd5"]
Sep 11 17:48:41.724096 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:41.724059    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:42.050942 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:42.050908    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e41b8708-23cf-409a-a8ed-ed8fba05c4bd" path="/var/lib/kubelet/pods/e41b8708-23cf-409a-a8ed-ed8fba05c4bd/volumes"
Sep 11 17:48:42.078493 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:42.078448    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:42.310713 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:42.310618    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:43.500256 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:43.500220    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:44.503487 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:44.503447    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:45.305789 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:45.305751    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:48:46.998136 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:46.998097    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"]
Sep 11 17:48:47.109525 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:48:47.109429    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:01.396669 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:01.396590    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:02.017448 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.017400    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746" podUID="48be2809-6027-471f-a140-44842e1e4af1" containerName="main" containerID="cri-o://467d517ecc9b2c46f044b60722eb8d6ede36df579568d0ce5b63b0038e71a392" gracePeriod=45
Sep 11 17:49:02.269574 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.269509    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:49:02.349595 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.349562    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-dshm\") pod \"48be2809-6027-471f-a140-44842e1e4af1\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") "
Sep 11 17:49:02.349764 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.349604    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw58l\" (UniqueName: \"kubernetes.io/projected/48be2809-6027-471f-a140-44842e1e4af1-kube-api-access-hw58l\") pod \"48be2809-6027-471f-a140-44842e1e4af1\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") "
Sep 11 17:49:02.349764 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.349622    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48be2809-6027-471f-a140-44842e1e4af1-tls-certs\") pod \"48be2809-6027-471f-a140-44842e1e4af1\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") "
Sep 11 17:49:02.349764 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.349679    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-tmp-dir\") pod \"48be2809-6027-471f-a140-44842e1e4af1\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") "
Sep 11 17:49:02.349764 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.349708    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-model-cache\") pod \"48be2809-6027-471f-a140-44842e1e4af1\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") "
Sep 11 17:49:02.349937 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.349775    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-home\") pod \"48be2809-6027-471f-a140-44842e1e4af1\" (UID: \"48be2809-6027-471f-a140-44842e1e4af1\") "
Sep 11 17:49:02.350147 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.350115    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-model-cache" (OuterVolumeSpecName: "model-cache") pod "48be2809-6027-471f-a140-44842e1e4af1" (UID: "48be2809-6027-471f-a140-44842e1e4af1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:49:02.350272 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.350187    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48be2809-6027-471f-a140-44842e1e4af1" (UID: "48be2809-6027-471f-a140-44842e1e4af1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:49:02.350330 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.350271    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-home" (OuterVolumeSpecName: "home") pod "48be2809-6027-471f-a140-44842e1e4af1" (UID: "48be2809-6027-471f-a140-44842e1e4af1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:49:02.352038 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.352011    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48be2809-6027-471f-a140-44842e1e4af1-kube-api-access-hw58l" (OuterVolumeSpecName: "kube-api-access-hw58l") pod "48be2809-6027-471f-a140-44842e1e4af1" (UID: "48be2809-6027-471f-a140-44842e1e4af1"). InnerVolumeSpecName "kube-api-access-hw58l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:49:02.352244 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.352222    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-dshm" (OuterVolumeSpecName: "dshm") pod "48be2809-6027-471f-a140-44842e1e4af1" (UID: "48be2809-6027-471f-a140-44842e1e4af1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:49:02.352321 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.352290    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48be2809-6027-471f-a140-44842e1e4af1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48be2809-6027-471f-a140-44842e1e4af1" (UID: "48be2809-6027-471f-a140-44842e1e4af1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:49:02.435015 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.434979    2560 generic.go:356] "Generic (PLEG): container finished" podID="48be2809-6027-471f-a140-44842e1e4af1" containerID="467d517ecc9b2c46f044b60722eb8d6ede36df579568d0ce5b63b0038e71a392" exitCode=0
Sep 11 17:49:02.435523 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.435027    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"
Sep 11 17:49:02.435523 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.435030    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746" event={"ID":"48be2809-6027-471f-a140-44842e1e4af1","Type":"ContainerDied","Data":"467d517ecc9b2c46f044b60722eb8d6ede36df579568d0ce5b63b0038e71a392"}
Sep 11 17:49:02.435523 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.435150    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746" event={"ID":"48be2809-6027-471f-a140-44842e1e4af1","Type":"ContainerDied","Data":"3cc08b6922ad5c0f2bfbc1a61fcf11051d5cddfe69099f68a89f5150d9581a64"}
Sep 11 17:49:02.435523 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.435173    2560 scope.go:117] "RemoveContainer" containerID="467d517ecc9b2c46f044b60722eb8d6ede36df579568d0ce5b63b0038e71a392"
Sep 11 17:49:02.445088 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.445069    2560 scope.go:117] "RemoveContainer" containerID="467d517ecc9b2c46f044b60722eb8d6ede36df579568d0ce5b63b0038e71a392"
Sep 11 17:49:02.445465 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:49:02.445443    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467d517ecc9b2c46f044b60722eb8d6ede36df579568d0ce5b63b0038e71a392\": container with ID starting with 467d517ecc9b2c46f044b60722eb8d6ede36df579568d0ce5b63b0038e71a392 not found: ID does not exist" containerID="467d517ecc9b2c46f044b60722eb8d6ede36df579568d0ce5b63b0038e71a392"
Sep 11 17:49:02.445536 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.445491    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467d517ecc9b2c46f044b60722eb8d6ede36df579568d0ce5b63b0038e71a392"} err="failed to get container status \"467d517ecc9b2c46f044b60722eb8d6ede36df579568d0ce5b63b0038e71a392\": rpc error: code = NotFound desc = could not find container \"467d517ecc9b2c46f044b60722eb8d6ede36df579568d0ce5b63b0038e71a392\": container with ID starting with 467d517ecc9b2c46f044b60722eb8d6ede36df579568d0ce5b63b0038e71a392 not found: ID does not exist"
Sep 11 17:49:02.451250 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.451232    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:02.451250 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.451251    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:02.451359 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.451259    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:02.451359 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.451268    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw58l\" (UniqueName: \"kubernetes.io/projected/48be2809-6027-471f-a140-44842e1e4af1-kube-api-access-hw58l\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:02.451359 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.451278    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48be2809-6027-471f-a140-44842e1e4af1-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:02.451359 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.451286    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48be2809-6027-471f-a140-44842e1e4af1-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:02.459120 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.459090    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"]
Sep 11 17:49:02.463098 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:02.463075    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-d8a0d0/ipp-route-d8a0d0-kserve-86499b649f-gp746"]
Sep 11 17:49:04.051322 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:04.051279    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48be2809-6027-471f-a140-44842e1e4af1" path="/var/lib/kubelet/pods/48be2809-6027-471f-a140-44842e1e4af1/volumes"
Sep 11 17:49:23.245870 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:23.245836    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:24.584387 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:24.584319    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:30.275354 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.275283    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:30.377075 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.377040    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:30.468323 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.468291    2560 scope.go:117] "RemoveContainer" containerID="29a555459f12fd9c0ac25088012144dca2e8a69558549da61369eb62c49910bd"
Sep 11 17:49:30.570779 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.570646    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:30.923114 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.923078    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b"]
Sep 11 17:49:30.923559 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.923541    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e41b8708-23cf-409a-a8ed-ed8fba05c4bd" containerName="revoke-keys"
Sep 11 17:49:30.923559 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.923557    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e41b8708-23cf-409a-a8ed-ed8fba05c4bd" containerName="revoke-keys"
Sep 11 17:49:30.923746 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.923571    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48be2809-6027-471f-a140-44842e1e4af1" containerName="main"
Sep 11 17:49:30.923746 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.923576    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="48be2809-6027-471f-a140-44842e1e4af1" containerName="main"
Sep 11 17:49:30.923746 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.923591    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4f308a-48e3-467c-813d-ced044138633" containerName="istio-proxy"
Sep 11 17:49:30.923746 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.923603    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4f308a-48e3-467c-813d-ced044138633" containerName="istio-proxy"
Sep 11 17:49:30.923746 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.923615    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4f308a-48e3-467c-813d-ced044138633" containerName="istio-proxy"
Sep 11 17:49:30.923746 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.923620    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4f308a-48e3-467c-813d-ced044138633" containerName="istio-proxy"
Sep 11 17:49:30.923746 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.923681    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4f308a-48e3-467c-813d-ced044138633" containerName="istio-proxy"
Sep 11 17:49:30.923746 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.923689    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e41b8708-23cf-409a-a8ed-ed8fba05c4bd" containerName="revoke-keys"
Sep 11 17:49:30.923746 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.923697    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4f308a-48e3-467c-813d-ced044138633" containerName="istio-proxy"
Sep 11 17:49:30.923746 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.923704    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="48be2809-6027-471f-a140-44842e1e4af1" containerName="main"
Sep 11 17:49:30.928084 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.928065    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b"
Sep 11 17:49:30.930825 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.930805    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:49:30.931314 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.931299    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:49:30.937453 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:30.937425    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b"]
Sep 11 17:49:31.023001 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:31.022958    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0068d5fc-60dc-4cfe-82df-5bf4b2c934da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b\" (UID: \"0068d5fc-60dc-4cfe-82df-5bf4b2c934da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b"
Sep 11 17:49:31.124090 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:31.124040    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0068d5fc-60dc-4cfe-82df-5bf4b2c934da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b\" (UID: \"0068d5fc-60dc-4cfe-82df-5bf4b2c934da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b"
Sep 11 17:49:31.124767 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:31.124741    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0068d5fc-60dc-4cfe-82df-5bf4b2c934da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b\" (UID: \"0068d5fc-60dc-4cfe-82df-5bf4b2c934da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b"
Sep 11 17:49:31.240531 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:31.240413    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b"
Sep 11 17:49:31.295515 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:31.295465    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:31.515296 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:31.515272    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b"]
Sep 11 17:49:31.517362 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:49:31.517334    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0068d5fc_60dc_4cfe_82df_5bf4b2c934da.slice/crio-678926860f2eea7d8ac3b0b527a77529ba1d5ba48c7ed606279f2483d052650e WatchSource:0}: Error finding container 678926860f2eea7d8ac3b0b527a77529ba1d5ba48c7ed606279f2483d052650e: Status 404 returned error can't find the container with id 678926860f2eea7d8ac3b0b527a77529ba1d5ba48c7ed606279f2483d052650e
Sep 11 17:49:31.592816 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:31.592782    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b" event={"ID":"0068d5fc-60dc-4cfe-82df-5bf4b2c934da","Type":"ContainerStarted","Data":"678926860f2eea7d8ac3b0b527a77529ba1d5ba48c7ed606279f2483d052650e"}
Sep 11 17:49:31.981072 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:31.981041    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:32.600239 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:32.600144    2560 generic.go:356] "Generic (PLEG): container finished" podID="0068d5fc-60dc-4cfe-82df-5bf4b2c934da" containerID="98248d965963af7758e8ccb2a2dbfcb109a3cd31ed2833bff319b9f89f4e3957" exitCode=0
Sep 11 17:49:32.600679 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:32.600237    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b" event={"ID":"0068d5fc-60dc-4cfe-82df-5bf4b2c934da","Type":"ContainerDied","Data":"98248d965963af7758e8ccb2a2dbfcb109a3cd31ed2833bff319b9f89f4e3957"}
Sep 11 17:49:32.785288 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:32.785252    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:33.626408 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:33.626369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:33.738413 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:33.738385    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b"
Sep 11 17:49:33.851222 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:33.851186    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0068d5fc-60dc-4cfe-82df-5bf4b2c934da-maas-api-service-ca\") pod \"0068d5fc-60dc-4cfe-82df-5bf4b2c934da\" (UID: \"0068d5fc-60dc-4cfe-82df-5bf4b2c934da\") "
Sep 11 17:49:33.851594 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:33.851569    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0068d5fc-60dc-4cfe-82df-5bf4b2c934da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0068d5fc-60dc-4cfe-82df-5bf4b2c934da" (UID: "0068d5fc-60dc-4cfe-82df-5bf4b2c934da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:49:33.952122 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:33.952080    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0068d5fc-60dc-4cfe-82df-5bf4b2c934da-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:34.334044 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:34.333955    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:34.612249 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:34.612167    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b"
Sep 11 17:49:34.612402 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:34.612166    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b" event={"ID":"0068d5fc-60dc-4cfe-82df-5bf4b2c934da","Type":"ContainerDied","Data":"678926860f2eea7d8ac3b0b527a77529ba1d5ba48c7ed606279f2483d052650e"}
Sep 11 17:49:34.612402 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:34.612300    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="678926860f2eea7d8ac3b0b527a77529ba1d5ba48c7ed606279f2483d052650e"
Sep 11 17:49:36.910749 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:36.910712    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:41.073855 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:41.073818    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:41.758017 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:41.757984    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:41.861770 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:41.861711    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:41.966192 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:41.966158    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:42.011430 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:42.011336    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b"]
Sep 11 17:49:42.013580 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:42.013550    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1d492f-mgh9b"]
Sep 11 17:49:42.051367 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:42.051319    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0068d5fc-60dc-4cfe-82df-5bf4b2c934da" path="/var/lib/kubelet/pods/0068d5fc-60dc-4cfe-82df-5bf4b2c934da/volumes"
Sep 11 17:49:42.068616 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:42.068582    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:42.377188 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:42.377067    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:42.480993 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:42.480959    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:43.473877 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:43.473845    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:44.399897 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:44.399836    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:48.609821 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:48.609765    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:48.685941 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:48.685907    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:48.865535 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:48.865433    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"]
Sep 11 17:49:48.865728 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:48.865695    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls" podUID="2472e12a-972b-4e94-9892-1a1ce97b8494" containerName="istio-proxy" containerID="cri-o://111f084aca00793834d030125d542d7c905368fd53506874bfc63bfe7e1151d3" gracePeriod=30
Sep 11 17:49:48.897097 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:48.897058    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:49.292753 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:49.292708    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7"]
Sep 11 17:49:49.293180 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:49.293158    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0068d5fc-60dc-4cfe-82df-5bf4b2c934da" containerName="revoke-keys"
Sep 11 17:49:49.293269 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:49.293182    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0068d5fc-60dc-4cfe-82df-5bf4b2c934da" containerName="revoke-keys"
Sep 11 17:49:49.293326 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:49.293308    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0068d5fc-60dc-4cfe-82df-5bf4b2c934da" containerName="revoke-keys"
Sep 11 17:49:49.296662 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:49.296637    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7"
Sep 11 17:49:49.300425 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:49.300403    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 17:49:49.300425 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:49.300419    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z84jg\""
Sep 11 17:49:49.321281 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:49.321246    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7"]
Sep 11 17:49:49.405781 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:49.405736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae8d6af5-e600-4302-8a18-870c092563bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7\" (UID: \"ae8d6af5-e600-4302-8a18-870c092563bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7"
Sep 11 17:49:49.467155 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:49.467123    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:49.506863 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:49.506823    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae8d6af5-e600-4302-8a18-870c092563bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7\" (UID: \"ae8d6af5-e600-4302-8a18-870c092563bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7"
Sep 11 17:49:49.507465 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:49.507444    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae8d6af5-e600-4302-8a18-870c092563bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7\" (UID: \"ae8d6af5-e600-4302-8a18-870c092563bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7"
Sep 11 17:49:49.607875 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:49.607775    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7"
Sep 11 17:49:49.881767 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:49.881740    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7"]
Sep 11 17:49:49.882748 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:49:49.882715    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8d6af5_e600_4302_8a18_870c092563bc.slice/crio-cf48a682eaa79f96d0f2c0ed525a820a30066ff70d8368758514b73969db05e1 WatchSource:0}: Error finding container cf48a682eaa79f96d0f2c0ed525a820a30066ff70d8368758514b73969db05e1: Status 404 returned error can't find the container with id cf48a682eaa79f96d0f2c0ed525a820a30066ff70d8368758514b73969db05e1
Sep 11 17:49:50.333363 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:50.333333    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:50.700440 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:50.700409    2560 generic.go:356] "Generic (PLEG): container finished" podID="ae8d6af5-e600-4302-8a18-870c092563bc" containerID="625919b60466b34524455e7cccfed1181e8b393e5c71b7853423dd993fc9c5d9" exitCode=0
Sep 11 17:49:50.700649 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:50.700505    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7" event={"ID":"ae8d6af5-e600-4302-8a18-870c092563bc","Type":"ContainerDied","Data":"625919b60466b34524455e7cccfed1181e8b393e5c71b7853423dd993fc9c5d9"}
Sep 11 17:49:50.700649 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:50.700564    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7" event={"ID":"ae8d6af5-e600-4302-8a18-870c092563bc","Type":"ContainerStarted","Data":"cf48a682eaa79f96d0f2c0ed525a820a30066ff70d8368758514b73969db05e1"}
Sep 11 17:49:51.241302 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:51.241262    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:51.843958 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:51.843931    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7"
Sep 11 17:49:51.933913 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:51.933868    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae8d6af5-e600-4302-8a18-870c092563bc-maas-api-service-ca\") pod \"ae8d6af5-e600-4302-8a18-870c092563bc\" (UID: \"ae8d6af5-e600-4302-8a18-870c092563bc\") "
Sep 11 17:49:51.934276 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:51.934252    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8d6af5-e600-4302-8a18-870c092563bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae8d6af5-e600-4302-8a18-870c092563bc" (UID: "ae8d6af5-e600-4302-8a18-870c092563bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:49:52.035440 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:52.035355    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae8d6af5-e600-4302-8a18-870c092563bc-maas-api-service-ca\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:52.055504 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:52.055445    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:52.713558 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:52.713527    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7"
Sep 11 17:49:52.713558 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:52.713545    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7" event={"ID":"ae8d6af5-e600-4302-8a18-870c092563bc","Type":"ContainerDied","Data":"cf48a682eaa79f96d0f2c0ed525a820a30066ff70d8368758514b73969db05e1"}
Sep 11 17:49:52.714006 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:52.713598    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf48a682eaa79f96d0f2c0ed525a820a30066ff70d8368758514b73969db05e1"
Sep 11 17:49:54.127142 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.127109    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:49:54.261666 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.261630    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-token\") pod \"2472e12a-972b-4e94-9892-1a1ce97b8494\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") "
Sep 11 17:49:54.261847 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.261693    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-data\") pod \"2472e12a-972b-4e94-9892-1a1ce97b8494\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") "
Sep 11 17:49:54.261847 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.261730    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-workload-certs\") pod \"2472e12a-972b-4e94-9892-1a1ce97b8494\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") "
Sep 11 17:49:54.261847 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.261780    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-podinfo\") pod \"2472e12a-972b-4e94-9892-1a1ce97b8494\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") "
Sep 11 17:49:54.261847 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.261805    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-envoy\") pod \"2472e12a-972b-4e94-9892-1a1ce97b8494\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") "
Sep 11 17:49:54.261847 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.261837    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-credential-socket\") pod \"2472e12a-972b-4e94-9892-1a1ce97b8494\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") "
Sep 11 17:49:54.262082 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.261863    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-workload-socket\") pod \"2472e12a-972b-4e94-9892-1a1ce97b8494\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") "
Sep 11 17:49:54.262082 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.261935    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jvfn\" (UniqueName: \"kubernetes.io/projected/2472e12a-972b-4e94-9892-1a1ce97b8494-kube-api-access-5jvfn\") pod \"2472e12a-972b-4e94-9892-1a1ce97b8494\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") "
Sep 11 17:49:54.262082 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.261965    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2472e12a-972b-4e94-9892-1a1ce97b8494-istiod-ca-cert\") pod \"2472e12a-972b-4e94-9892-1a1ce97b8494\" (UID: \"2472e12a-972b-4e94-9892-1a1ce97b8494\") "
Sep 11 17:49:54.262082 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.262043    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2472e12a-972b-4e94-9892-1a1ce97b8494" (UID: "2472e12a-972b-4e94-9892-1a1ce97b8494"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:49:54.262082 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.262059    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-data" (OuterVolumeSpecName: "istio-data") pod "2472e12a-972b-4e94-9892-1a1ce97b8494" (UID: "2472e12a-972b-4e94-9892-1a1ce97b8494"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:49:54.262365 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.262159    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2472e12a-972b-4e94-9892-1a1ce97b8494" (UID: "2472e12a-972b-4e94-9892-1a1ce97b8494"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:49:54.262365 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.262209    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2472e12a-972b-4e94-9892-1a1ce97b8494" (UID: "2472e12a-972b-4e94-9892-1a1ce97b8494"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:49:54.262505 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.262369    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-data\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:54.262505 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.262383    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2472e12a-972b-4e94-9892-1a1ce97b8494-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2472e12a-972b-4e94-9892-1a1ce97b8494" (UID: "2472e12a-972b-4e94-9892-1a1ce97b8494"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:49:54.262505 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.262390    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-workload-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:54.262505 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.262415    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-credential-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:54.262505 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.262430    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-workload-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:54.264164 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.264140    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2472e12a-972b-4e94-9892-1a1ce97b8494" (UID: "2472e12a-972b-4e94-9892-1a1ce97b8494"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:49:54.264164 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.264150    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2472e12a-972b-4e94-9892-1a1ce97b8494-kube-api-access-5jvfn" (OuterVolumeSpecName: "kube-api-access-5jvfn") pod "2472e12a-972b-4e94-9892-1a1ce97b8494" (UID: "2472e12a-972b-4e94-9892-1a1ce97b8494"). InnerVolumeSpecName "kube-api-access-5jvfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:49:54.264343 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.264326    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-token" (OuterVolumeSpecName: "istio-token") pod "2472e12a-972b-4e94-9892-1a1ce97b8494" (UID: "2472e12a-972b-4e94-9892-1a1ce97b8494"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:49:54.264385 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.264353    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2472e12a-972b-4e94-9892-1a1ce97b8494" (UID: "2472e12a-972b-4e94-9892-1a1ce97b8494"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:49:54.363773 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.363734    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-podinfo\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:54.363773 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.363765    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-envoy\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:54.363773 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.363777    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jvfn\" (UniqueName: \"kubernetes.io/projected/2472e12a-972b-4e94-9892-1a1ce97b8494-kube-api-access-5jvfn\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:54.363773 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.363787    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2472e12a-972b-4e94-9892-1a1ce97b8494-istiod-ca-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:54.364054 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.363796    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2472e12a-972b-4e94-9892-1a1ce97b8494-istio-token\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:49:54.725424 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.725387    2560 generic.go:356] "Generic (PLEG): container finished" podID="2472e12a-972b-4e94-9892-1a1ce97b8494" containerID="111f084aca00793834d030125d542d7c905368fd53506874bfc63bfe7e1151d3" exitCode=0
Sep 11 17:49:54.725668 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.725442    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"
Sep 11 17:49:54.725668 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.725453    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls" event={"ID":"2472e12a-972b-4e94-9892-1a1ce97b8494","Type":"ContainerDied","Data":"111f084aca00793834d030125d542d7c905368fd53506874bfc63bfe7e1151d3"}
Sep 11 17:49:54.725668 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.725517    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls" event={"ID":"2472e12a-972b-4e94-9892-1a1ce97b8494","Type":"ContainerDied","Data":"fadc7504d25d3e6f8d3039ddb3052ae97883ab29ec3de9c2f736d14eda0b460f"}
Sep 11 17:49:54.725668 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.725548    2560 scope.go:117] "RemoveContainer" containerID="111f084aca00793834d030125d542d7c905368fd53506874bfc63bfe7e1151d3"
Sep 11 17:49:54.735110 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.735089    2560 scope.go:117] "RemoveContainer" containerID="f3d20ea077bb68a1cfa11cd024e66ddbd43f445b6677c6006602734a533e46d4"
Sep 11 17:49:54.743986 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.743963    2560 scope.go:117] "RemoveContainer" containerID="111f084aca00793834d030125d542d7c905368fd53506874bfc63bfe7e1151d3"
Sep 11 17:49:54.744316 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:49:54.744291    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111f084aca00793834d030125d542d7c905368fd53506874bfc63bfe7e1151d3\": container with ID starting with 111f084aca00793834d030125d542d7c905368fd53506874bfc63bfe7e1151d3 not found: ID does not exist" containerID="111f084aca00793834d030125d542d7c905368fd53506874bfc63bfe7e1151d3"
Sep 11 17:49:54.744397 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.744325    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111f084aca00793834d030125d542d7c905368fd53506874bfc63bfe7e1151d3"} err="failed to get container status \"111f084aca00793834d030125d542d7c905368fd53506874bfc63bfe7e1151d3\": rpc error: code = NotFound desc = could not find container \"111f084aca00793834d030125d542d7c905368fd53506874bfc63bfe7e1151d3\": container with ID starting with 111f084aca00793834d030125d542d7c905368fd53506874bfc63bfe7e1151d3 not found: ID does not exist"
Sep 11 17:49:54.744397 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.744344    2560 scope.go:117] "RemoveContainer" containerID="f3d20ea077bb68a1cfa11cd024e66ddbd43f445b6677c6006602734a533e46d4"
Sep 11 17:49:54.745099 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:49:54.744731    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3d20ea077bb68a1cfa11cd024e66ddbd43f445b6677c6006602734a533e46d4\": container with ID starting with f3d20ea077bb68a1cfa11cd024e66ddbd43f445b6677c6006602734a533e46d4 not found: ID does not exist" containerID="f3d20ea077bb68a1cfa11cd024e66ddbd43f445b6677c6006602734a533e46d4"
Sep 11 17:49:54.745099 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.744759    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d20ea077bb68a1cfa11cd024e66ddbd43f445b6677c6006602734a533e46d4"} err="failed to get container status \"f3d20ea077bb68a1cfa11cd024e66ddbd43f445b6677c6006602734a533e46d4\": rpc error: code = NotFound desc = could not find container \"f3d20ea077bb68a1cfa11cd024e66ddbd43f445b6677c6006602734a533e46d4\": container with ID starting with f3d20ea077bb68a1cfa11cd024e66ddbd43f445b6677c6006602734a533e46d4 not found: ID does not exist"
Sep 11 17:49:54.754430 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.754402    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"]
Sep 11 17:49:54.760063 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:54.760035    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1d492f-openshift-default-7fd9dd7fc6-9bzls"]
Sep 11 17:49:56.051071 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:56.051037    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2472e12a-972b-4e94-9892-1a1ce97b8494" path="/var/lib/kubelet/pods/2472e12a-972b-4e94-9892-1a1ce97b8494/volumes"
Sep 11 17:49:59.882749 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:59.882712    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4c36f4-29819145-85ph6"]
Sep 11 17:49:59.887232 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:59.887198    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-4c36f4-29819145-85ph6"]
Sep 11 17:49:59.892582 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:59.892552    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:49:59.963324 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:49:59.963293    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:00.050755 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:00.050726    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c2db8bb-6fa7-428e-9edd-6e132af5ff50" path="/var/lib/kubelet/pods/1c2db8bb-6fa7-428e-9edd-6e132af5ff50/volumes"
Sep 11 17:50:00.064047 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:00.064017    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:00.113877 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:00.113840    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7"]
Sep 11 17:50:00.116185 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:00.116156    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-4c36f4-95ph7"]
Sep 11 17:50:00.149425 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:00.149318    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:00.492986 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:00.492900    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:00.601566 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:00.601537    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:01.326910 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:01.326876    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:02.051529 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:02.051496    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae8d6af5-e600-4302-8a18-870c092563bc" path="/var/lib/kubelet/pods/ae8d6af5-e600-4302-8a18-870c092563bc/volumes"
Sep 11 17:50:06.735213 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:06.735182    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:06.809486 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:06.809448    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:07.023267 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:07.023181    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:07.668694 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:07.668652    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:08.464562 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:08.464529    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:09.359416 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:09.359387    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:17.983798 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:17.983762    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:17.988523 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:17.988498    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db95fd-29819145-9vxjg"]
Sep 11 17:50:17.991514 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:17.991469    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-db95fd-29819145-9vxjg"]
Sep 11 17:50:18.051785 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:18.051751    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0e92682-44d3-45b5-b0cf-fee6e3255351" path="/var/lib/kubelet/pods/b0e92682-44d3-45b5-b0cf-fee6e3255351/volumes"
Sep 11 17:50:18.110608 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:18.110571    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:18.243714 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:18.243624    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:18.368764 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:18.368727    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:18.470625 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:18.470593    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:18.810962 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:18.810931    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:18.976681 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:18.976646    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:25.084748 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:25.084667    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:25.186190 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:25.186146    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:25.363649 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:25.363560    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"]
Sep 11 17:50:25.363841 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:25.363776    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx" podUID="ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" containerName="istio-proxy" containerID="cri-o://b5586f314324ab2aa957f7bc43b9155da3eb38d003704081edcf9feabfed3127" gracePeriod=30
Sep 11 17:50:25.422326 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:25.422274    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:50:30.134141 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.134111    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:50:30.138097 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.138069    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:50:30.499242 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.499215    2560 scope.go:117] "RemoveContainer" containerID="a64c51287d018cc49908f36120e97fc3ae602f49cb728ae86e09defe1cfa8d5c"
Sep 11 17:50:30.509601 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.509579    2560 scope.go:117] "RemoveContainer" containerID="c100da3759dc6ad995e2cc6c368d113d31d9e59e163a4655cb6a15aa17380d16"
Sep 11 17:50:30.520052 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.520029    2560 scope.go:117] "RemoveContainer" containerID="510e99a01ed52d6c29cae813d2ff0ce2d5ee416a0a4364e014eaf108a7954e2a"
Sep 11 17:50:30.628947 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.628918    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:50:30.644092 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.643997    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-workload-socket\") pod \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") "
Sep 11 17:50:30.644092 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.644046    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-token\") pod \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") "
Sep 11 17:50:30.644338 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.644109    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istiod-ca-cert\") pod \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") "
Sep 11 17:50:30.644338 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.644135    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgh66\" (UniqueName: \"kubernetes.io/projected/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-kube-api-access-kgh66\") pod \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") "
Sep 11 17:50:30.644338 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.644262    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" (UID: "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:30.644338 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.644308    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-credential-socket\") pod \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") "
Sep 11 17:50:30.644582 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.644341    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-workload-certs\") pod \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") "
Sep 11 17:50:30.644582 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.644439    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-envoy\") pod \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") "
Sep 11 17:50:30.644582 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.644499    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-podinfo\") pod \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") "
Sep 11 17:50:30.644582 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.644570    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" (UID: "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:50:30.644821 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.644584    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-data\") pod \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\" (UID: \"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202\") "
Sep 11 17:50:30.645022 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.644936    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-workload-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:50:30.645022 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.644960    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istiod-ca-cert\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:50:30.645194 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.645131    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" (UID: "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:30.645194 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.645151    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" (UID: "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:30.645415 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.645392    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-data" (OuterVolumeSpecName: "istio-data") pod "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" (UID: "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:30.646626 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.646603    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-kube-api-access-kgh66" (OuterVolumeSpecName: "kube-api-access-kgh66") pod "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" (UID: "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202"). InnerVolumeSpecName "kube-api-access-kgh66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:50:30.647215 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.647194    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-token" (OuterVolumeSpecName: "istio-token") pod "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" (UID: "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:50:30.647569 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.647546    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" (UID: "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:50:30.647569 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.647553    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" (UID: "ff3cfdc6-dc6b-4fe2-8896-42b869d9e202"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:50:30.745868 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.745826    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-envoy\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:50:30.745868 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.745861    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-podinfo\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:50:30.745868 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.745870    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-data\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:50:30.745868 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.745879    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-istio-token\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:50:30.746159 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.745887    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgh66\" (UniqueName: \"kubernetes.io/projected/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-kube-api-access-kgh66\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:50:30.746159 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.745896    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-credential-socket\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:50:30.746159 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.745904    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202-workload-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:50:30.916200 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.916111    2560 generic.go:356] "Generic (PLEG): container finished" podID="ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" containerID="b5586f314324ab2aa957f7bc43b9155da3eb38d003704081edcf9feabfed3127" exitCode=0
Sep 11 17:50:30.916200 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.916179    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"
Sep 11 17:50:30.916384 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.916201    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx" event={"ID":"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202","Type":"ContainerDied","Data":"b5586f314324ab2aa957f7bc43b9155da3eb38d003704081edcf9feabfed3127"}
Sep 11 17:50:30.916384 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.916296    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx" event={"ID":"ff3cfdc6-dc6b-4fe2-8896-42b869d9e202","Type":"ContainerDied","Data":"05dfae5da28bd3404420a1f6078c02f261bec45f5424295c1c41ec7ee01d6b78"}
Sep 11 17:50:30.916384 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.916324    2560 scope.go:117] "RemoveContainer" containerID="b5586f314324ab2aa957f7bc43b9155da3eb38d003704081edcf9feabfed3127"
Sep 11 17:50:30.925683 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.925664    2560 scope.go:117] "RemoveContainer" containerID="b5586f314324ab2aa957f7bc43b9155da3eb38d003704081edcf9feabfed3127"
Sep 11 17:50:30.926052 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:50:30.926023    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5586f314324ab2aa957f7bc43b9155da3eb38d003704081edcf9feabfed3127\": container with ID starting with b5586f314324ab2aa957f7bc43b9155da3eb38d003704081edcf9feabfed3127 not found: ID does not exist" containerID="b5586f314324ab2aa957f7bc43b9155da3eb38d003704081edcf9feabfed3127"
Sep 11 17:50:30.926197 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.926057    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5586f314324ab2aa957f7bc43b9155da3eb38d003704081edcf9feabfed3127"} err="failed to get container status \"b5586f314324ab2aa957f7bc43b9155da3eb38d003704081edcf9feabfed3127\": rpc error: code = NotFound desc = could not find container \"b5586f314324ab2aa957f7bc43b9155da3eb38d003704081edcf9feabfed3127\": container with ID starting with b5586f314324ab2aa957f7bc43b9155da3eb38d003704081edcf9feabfed3127 not found: ID does not exist"
Sep 11 17:50:30.951934 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.951898    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"]
Sep 11 17:50:30.956881 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:30.956844    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db95fd-openshift-default-6fcbcd59db-lxnkx"]
Sep 11 17:50:32.050719 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:50:32.050685    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" path="/var/lib/kubelet/pods/ff3cfdc6-dc6b-4fe2-8896-42b869d9e202/volumes"
Sep 11 17:51:21.108894 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:51:21.108858    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:51:22.432219 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:51:22.432182    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:51:30.562926 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:51:30.562890    2560 scope.go:117] "RemoveContainer" containerID="040240859fe35a9ccc36b71144ffecf971c0eb774d2bc4b61d7179077c3a34b2"
Sep 11 17:51:30.571521 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:51:30.571494    2560 scope.go:117] "RemoveContainer" containerID="33eb4d9c3ffaf91e2e23fca6f83cdfb73306f226447960303368d170a1aeb8fe"
Sep 11 17:51:30.579920 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:51:30.579896    2560 scope.go:117] "RemoveContainer" containerID="a94d469e5e02423e62a109df4f509da2f78523e63ed4bb2b20b91ee5483ac7e0"
Sep 11 17:51:48.149068 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:51:48.149031    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:51:48.226727 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:51:48.226692    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:51:53.503397 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:51:53.503314    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:51:54.909951 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:51:54.909919    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:52:38.820742 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:52:38.820703    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:53:05.385413 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:53:05.385371    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:53:05.482392 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:53:05.482331    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:54:30.629091 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:30.629052    2560 scope.go:117] "RemoveContainer" containerID="78c62faaa362ae77d626f7ccacf7b123d737e9f2bafc6f299de2bb668a387b1b"
Sep 11 17:54:30.638004 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:30.637976    2560 scope.go:117] "RemoveContainer" containerID="144ab59dc91e6f01b49507db60c655f0c364c5a8f091ea4e6d4f36ebbd57e2b4"
Sep 11 17:54:55.630638 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.630556    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"]
Sep 11 17:54:55.631115 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.631099    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" containerName="istio-proxy"
Sep 11 17:54:55.631164 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.631116    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" containerName="istio-proxy"
Sep 11 17:54:55.631164 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.631132    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2472e12a-972b-4e94-9892-1a1ce97b8494" containerName="istio-proxy"
Sep 11 17:54:55.631164 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.631137    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2472e12a-972b-4e94-9892-1a1ce97b8494" containerName="istio-proxy"
Sep 11 17:54:55.631164 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.631144    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" containerName="istio-proxy"
Sep 11 17:54:55.631164 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.631151    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" containerName="istio-proxy"
Sep 11 17:54:55.631164 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.631159    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2472e12a-972b-4e94-9892-1a1ce97b8494" containerName="istio-proxy"
Sep 11 17:54:55.631164 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.631164    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2472e12a-972b-4e94-9892-1a1ce97b8494" containerName="istio-proxy"
Sep 11 17:54:55.631379 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.631179    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae8d6af5-e600-4302-8a18-870c092563bc" containerName="revoke-keys"
Sep 11 17:54:55.631379 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.631184    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae8d6af5-e600-4302-8a18-870c092563bc" containerName="revoke-keys"
Sep 11 17:54:55.631379 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.631242    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2472e12a-972b-4e94-9892-1a1ce97b8494" containerName="istio-proxy"
Sep 11 17:54:55.631379 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.631251    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2472e12a-972b-4e94-9892-1a1ce97b8494" containerName="istio-proxy"
Sep 11 17:54:55.631379 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.631258    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" containerName="istio-proxy"
Sep 11 17:54:55.631379 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.631265    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae8d6af5-e600-4302-8a18-870c092563bc" containerName="revoke-keys"
Sep 11 17:54:55.634508 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.634483    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.637335 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.637314    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e145a56e-kserve-self-signed-certs\""
Sep 11 17:54:55.646299 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.646274    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"]
Sep 11 17:54:55.721867 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.721833    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:54:55.728493 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.728445    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-tmp-dir\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.728680 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.728549    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd9qf\" (UniqueName: \"kubernetes.io/projected/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-kube-api-access-jd9qf\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.728680 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.728601    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-model-cache\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.728680 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.728626    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-home\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.728816 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.728747    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-tls-certs\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.728816 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.728779    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-dshm\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.829688 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.829651    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-model-cache\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.829863 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.829694    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-home\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.829863 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.829756    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-tls-certs\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.829863 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.829778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-dshm\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.830050 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.829871    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-tmp-dir\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.830050 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.829900    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd9qf\" (UniqueName: \"kubernetes.io/projected/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-kube-api-access-jd9qf\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.830148 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.830099    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-model-cache\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.830200 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.830139    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-home\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.830272 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.830252    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-tmp-dir\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.832072 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.832047    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-dshm\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.832331 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.832315    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-tls-certs\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.840146 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.840115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd9qf\" (UniqueName: \"kubernetes.io/projected/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-kube-api-access-jd9qf\") pod \"e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") " pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:55.944525 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:55.944399    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:54:56.238924 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:56.238898    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"]
Sep 11 17:54:56.240110 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:54:56.240085    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67e21c2f_4429_46d1_aff8_b58e6cbbb24d.slice/crio-6e978d1c01ab14219d3b3bc6d4488e87120fdca9ec85edaa54fc40f958691f0a WatchSource:0}: Error finding container 6e978d1c01ab14219d3b3bc6d4488e87120fdca9ec85edaa54fc40f958691f0a: Status 404 returned error can't find the container with id 6e978d1c01ab14219d3b3bc6d4488e87120fdca9ec85edaa54fc40f958691f0a
Sep 11 17:54:56.262871 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:56.262846    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95" event={"ID":"67e21c2f-4429-46d1-aff8-b58e6cbbb24d","Type":"ContainerStarted","Data":"6e978d1c01ab14219d3b3bc6d4488e87120fdca9ec85edaa54fc40f958691f0a"}
Sep 11 17:54:57.270074 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:57.270035    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95" event={"ID":"67e21c2f-4429-46d1-aff8-b58e6cbbb24d","Type":"ContainerStarted","Data":"8bc3ae1b6b071028302d5f847a59433717656f10f9a25dcf99e1b8a0dc374c3d"}
Sep 11 17:54:57.292300 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:54:57.292248    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95" podStartSLOduration=2.29223082 podStartE2EDuration="2.29223082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:57.289628481 +0000 UTC m=+2067.722081268" watchObservedRunningTime="2026-09-11 17:54:57.29223082 +0000 UTC m=+2067.724683597"
Sep 11 17:55:05.945527 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:05.945455    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:55:05.945937 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:05.945542    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:55:05.958264 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:05.958228    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:55:06.326537 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:06.326424    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:55:09.953741 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:09.953697    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:55:10.016774 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:10.016739    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:55:14.135771 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.135733    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"]
Sep 11 17:55:14.136461 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.136439    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff3cfdc6-dc6b-4fe2-8896-42b869d9e202" containerName="istio-proxy"
Sep 11 17:55:14.139961 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.139934    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.146571 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.146544    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e145a56e-kserve-self-signed-certs\""
Sep 11 17:55:14.150784 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.150758    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"]
Sep 11 17:55:14.159913 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.159874    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:55:14.208715 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.208675    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-tmp-dir\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.208904 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.208737    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-model-cache\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.208904 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.208811    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cnt7\" (UniqueName: \"kubernetes.io/projected/e0b0a00d-305a-4580-b243-948cab628790-kube-api-access-8cnt7\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.208904 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.208858    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-dshm\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.208904 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.208890    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-home\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.209069 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.208911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b0a00d-305a-4580-b243-948cab628790-tls-certs\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.309617 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.309574    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-dshm\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.309805 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.309628    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-home\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.309805 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.309652    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b0a00d-305a-4580-b243-948cab628790-tls-certs\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.309805 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.309712    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-tmp-dir\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.309805 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.309753    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-model-cache\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.309805 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.309783    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cnt7\" (UniqueName: \"kubernetes.io/projected/e0b0a00d-305a-4580-b243-948cab628790-kube-api-access-8cnt7\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.310142 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.310118    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-home\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.310234 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.310163    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-tmp-dir\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.310234 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.310192    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-model-cache\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.312110 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.312088    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-dshm\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.312252 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.312237    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b0a00d-305a-4580-b243-948cab628790-tls-certs\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.319147 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.319109    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cnt7\" (UniqueName: \"kubernetes.io/projected/e0b0a00d-305a-4580-b243-948cab628790-kube-api-access-8cnt7\") pod \"e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") " pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.452046 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.451946    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:14.730306 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:14.730276    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"]
Sep 11 17:55:14.732090 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:55:14.732056    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0b0a00d_305a_4580_b243_948cab628790.slice/crio-a84de8fb7187748c1be1684f8a5aa651bea9d004a417a6d95d8e998ce37870f4 WatchSource:0}: Error finding container a84de8fb7187748c1be1684f8a5aa651bea9d004a417a6d95d8e998ce37870f4: Status 404 returned error can't find the container with id a84de8fb7187748c1be1684f8a5aa651bea9d004a417a6d95d8e998ce37870f4
Sep 11 17:55:15.359278 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:15.359239    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27" event={"ID":"e0b0a00d-305a-4580-b243-948cab628790","Type":"ContainerStarted","Data":"1b2c25b80bcf8eef3fffeeeb712bdd6618cdb3e86e526b13db190ff048c4ec6f"}
Sep 11 17:55:15.359687 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:15.359289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27" event={"ID":"e0b0a00d-305a-4580-b243-948cab628790","Type":"ContainerStarted","Data":"a84de8fb7187748c1be1684f8a5aa651bea9d004a417a6d95d8e998ce37870f4"}
Sep 11 17:55:15.382141 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:15.382090    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27" podStartSLOduration=1.382072853 podStartE2EDuration="1.382072853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:15.380975934 +0000 UTC m=+2085.813428712" watchObservedRunningTime="2026-09-11 17:55:15.382072853 +0000 UTC m=+2085.814525629"
Sep 11 17:55:24.452271 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:24.452234    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:24.452719 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:24.452308    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:24.465192 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:24.465161    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:25.426346 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:25.426312    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:30.165783 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:30.165755    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:55:30.176793 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:30.176767    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdw6t_1263d2ec-eaf2-44a9-b5db-64955c9b0f2f/ovn-acl-logging/0.log"
Sep 11 17:55:35.044048 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:35.044009    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"]
Sep 11 17:55:35.055392 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:35.055360    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:55:36.830062 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:36.830023    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"]
Sep 11 17:55:36.854422 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:36.854389    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:55:49.321966 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:49.321929    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:55:50.063303 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.063268    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95" podUID="67e21c2f-4429-46d1-aff8-b58e6cbbb24d" containerName="main" containerID="cri-o://8bc3ae1b6b071028302d5f847a59433717656f10f9a25dcf99e1b8a0dc374c3d" gracePeriod=45
Sep 11 17:55:50.323260 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.323188    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:55:50.478194 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.478156    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-dshm\") pod \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") "
Sep 11 17:55:50.478194 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.478201    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jd9qf\" (UniqueName: \"kubernetes.io/projected/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-kube-api-access-jd9qf\") pod \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") "
Sep 11 17:55:50.478461 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.478264    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-tmp-dir\") pod \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") "
Sep 11 17:55:50.478461 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.478297    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-home\") pod \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") "
Sep 11 17:55:50.478461 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.478320    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-model-cache\") pod \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") "
Sep 11 17:55:50.478461 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.478374    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-tls-certs\") pod \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\" (UID: \"67e21c2f-4429-46d1-aff8-b58e6cbbb24d\") "
Sep 11 17:55:50.478715 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.478567    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-home" (OuterVolumeSpecName: "home") pod "67e21c2f-4429-46d1-aff8-b58e6cbbb24d" (UID: "67e21c2f-4429-46d1-aff8-b58e6cbbb24d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:55:50.478715 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.478583    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67e21c2f-4429-46d1-aff8-b58e6cbbb24d" (UID: "67e21c2f-4429-46d1-aff8-b58e6cbbb24d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:55:50.478715 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.478680    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-model-cache" (OuterVolumeSpecName: "model-cache") pod "67e21c2f-4429-46d1-aff8-b58e6cbbb24d" (UID: "67e21c2f-4429-46d1-aff8-b58e6cbbb24d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:55:50.478836 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.478757    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:55:50.478836 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.478773    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:55:50.480653 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.480623    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67e21c2f-4429-46d1-aff8-b58e6cbbb24d" (UID: "67e21c2f-4429-46d1-aff8-b58e6cbbb24d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:55:50.480774 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.480658    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-dshm" (OuterVolumeSpecName: "dshm") pod "67e21c2f-4429-46d1-aff8-b58e6cbbb24d" (UID: "67e21c2f-4429-46d1-aff8-b58e6cbbb24d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:55:50.480774 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.480698    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-kube-api-access-jd9qf" (OuterVolumeSpecName: "kube-api-access-jd9qf") pod "67e21c2f-4429-46d1-aff8-b58e6cbbb24d" (UID: "67e21c2f-4429-46d1-aff8-b58e6cbbb24d"). InnerVolumeSpecName "kube-api-access-jd9qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:55:50.537194 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.537151    2560 generic.go:356] "Generic (PLEG): container finished" podID="67e21c2f-4429-46d1-aff8-b58e6cbbb24d" containerID="8bc3ae1b6b071028302d5f847a59433717656f10f9a25dcf99e1b8a0dc374c3d" exitCode=0
Sep 11 17:55:50.537194 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.537189    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"
Sep 11 17:55:50.537425 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.537214    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95" event={"ID":"67e21c2f-4429-46d1-aff8-b58e6cbbb24d","Type":"ContainerDied","Data":"8bc3ae1b6b071028302d5f847a59433717656f10f9a25dcf99e1b8a0dc374c3d"}
Sep 11 17:55:50.537425 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.537252    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95" event={"ID":"67e21c2f-4429-46d1-aff8-b58e6cbbb24d","Type":"ContainerDied","Data":"6e978d1c01ab14219d3b3bc6d4488e87120fdca9ec85edaa54fc40f958691f0a"}
Sep 11 17:55:50.537425 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.537272    2560 scope.go:117] "RemoveContainer" containerID="8bc3ae1b6b071028302d5f847a59433717656f10f9a25dcf99e1b8a0dc374c3d"
Sep 11 17:55:50.546950 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.546931    2560 scope.go:117] "RemoveContainer" containerID="8bc3ae1b6b071028302d5f847a59433717656f10f9a25dcf99e1b8a0dc374c3d"
Sep 11 17:55:50.547242 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:55:50.547197    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc3ae1b6b071028302d5f847a59433717656f10f9a25dcf99e1b8a0dc374c3d\": container with ID starting with 8bc3ae1b6b071028302d5f847a59433717656f10f9a25dcf99e1b8a0dc374c3d not found: ID does not exist" containerID="8bc3ae1b6b071028302d5f847a59433717656f10f9a25dcf99e1b8a0dc374c3d"
Sep 11 17:55:50.547242 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.547227    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc3ae1b6b071028302d5f847a59433717656f10f9a25dcf99e1b8a0dc374c3d"} err="failed to get container status \"8bc3ae1b6b071028302d5f847a59433717656f10f9a25dcf99e1b8a0dc374c3d\": rpc error: code = NotFound desc = could not find container \"8bc3ae1b6b071028302d5f847a59433717656f10f9a25dcf99e1b8a0dc374c3d\": container with ID starting with 8bc3ae1b6b071028302d5f847a59433717656f10f9a25dcf99e1b8a0dc374c3d not found: ID does not exist"
Sep 11 17:55:50.564160 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.564133    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"]
Sep 11 17:55:50.575428 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.575348    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e145a56e-kserve-7854f8c4d7-d7n95"]
Sep 11 17:55:50.579392 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.579370    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:55:50.579392 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.579393    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:55:50.579578 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.579406    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:55:50.579578 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:50.579419    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd9qf\" (UniqueName: \"kubernetes.io/projected/67e21c2f-4429-46d1-aff8-b58e6cbbb24d-kube-api-access-jd9qf\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:55:51.849359 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:51.849317    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27" podUID="e0b0a00d-305a-4580-b243-948cab628790" containerName="main" containerID="cri-o://1b2c25b80bcf8eef3fffeeeb712bdd6618cdb3e86e526b13db190ff048c4ec6f" gracePeriod=45
Sep 11 17:55:52.051676 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.051643    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e21c2f-4429-46d1-aff8-b58e6cbbb24d" path="/var/lib/kubelet/pods/67e21c2f-4429-46d1-aff8-b58e6cbbb24d/volumes"
Sep 11 17:55:52.097602 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.097576    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:52.194155 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.194103    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b0a00d-305a-4580-b243-948cab628790-tls-certs\") pod \"e0b0a00d-305a-4580-b243-948cab628790\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") "
Sep 11 17:55:52.194380 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.194192    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cnt7\" (UniqueName: \"kubernetes.io/projected/e0b0a00d-305a-4580-b243-948cab628790-kube-api-access-8cnt7\") pod \"e0b0a00d-305a-4580-b243-948cab628790\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") "
Sep 11 17:55:52.194380 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.194230    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-home\") pod \"e0b0a00d-305a-4580-b243-948cab628790\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") "
Sep 11 17:55:52.194380 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.194287    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-tmp-dir\") pod \"e0b0a00d-305a-4580-b243-948cab628790\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") "
Sep 11 17:55:52.194380 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.194312    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-model-cache\") pod \"e0b0a00d-305a-4580-b243-948cab628790\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") "
Sep 11 17:55:52.194380 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.194330    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-dshm\") pod \"e0b0a00d-305a-4580-b243-948cab628790\" (UID: \"e0b0a00d-305a-4580-b243-948cab628790\") "
Sep 11 17:55:52.194736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.194621    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0b0a00d-305a-4580-b243-948cab628790" (UID: "e0b0a00d-305a-4580-b243-948cab628790"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:55:52.194736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.194634    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-home" (OuterVolumeSpecName: "home") pod "e0b0a00d-305a-4580-b243-948cab628790" (UID: "e0b0a00d-305a-4580-b243-948cab628790"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:55:52.194736 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.194650    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0b0a00d-305a-4580-b243-948cab628790" (UID: "e0b0a00d-305a-4580-b243-948cab628790"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:55:52.196551 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.196520    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-dshm" (OuterVolumeSpecName: "dshm") pod "e0b0a00d-305a-4580-b243-948cab628790" (UID: "e0b0a00d-305a-4580-b243-948cab628790"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:55:52.196648 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.196522    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b0a00d-305a-4580-b243-948cab628790-kube-api-access-8cnt7" (OuterVolumeSpecName: "kube-api-access-8cnt7") pod "e0b0a00d-305a-4580-b243-948cab628790" (UID: "e0b0a00d-305a-4580-b243-948cab628790"). InnerVolumeSpecName "kube-api-access-8cnt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:55:52.196648 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.196593    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0b0a00d-305a-4580-b243-948cab628790-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0b0a00d-305a-4580-b243-948cab628790" (UID: "e0b0a00d-305a-4580-b243-948cab628790"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:55:52.295388 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.295339    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cnt7\" (UniqueName: \"kubernetes.io/projected/e0b0a00d-305a-4580-b243-948cab628790-kube-api-access-8cnt7\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:55:52.295388 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.295381    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-home\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:55:52.295388 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.295395    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-tmp-dir\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:55:52.295688 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.295408    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-model-cache\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:55:52.295688 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.295418    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0b0a00d-305a-4580-b243-948cab628790-dshm\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:55:52.295688 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.295428    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0b0a00d-305a-4580-b243-948cab628790-tls-certs\") on node \"ip-10-0-138-148.ec2.internal\" DevicePath \"\""
Sep 11 17:55:52.549320 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.549232    2560 generic.go:356] "Generic (PLEG): container finished" podID="e0b0a00d-305a-4580-b243-948cab628790" containerID="1b2c25b80bcf8eef3fffeeeb712bdd6618cdb3e86e526b13db190ff048c4ec6f" exitCode=0
Sep 11 17:55:52.549320 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.549274    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"
Sep 11 17:55:52.549559 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.549315    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27" event={"ID":"e0b0a00d-305a-4580-b243-948cab628790","Type":"ContainerDied","Data":"1b2c25b80bcf8eef3fffeeeb712bdd6618cdb3e86e526b13db190ff048c4ec6f"}
Sep 11 17:55:52.549559 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.549364    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27" event={"ID":"e0b0a00d-305a-4580-b243-948cab628790","Type":"ContainerDied","Data":"a84de8fb7187748c1be1684f8a5aa651bea9d004a417a6d95d8e998ce37870f4"}
Sep 11 17:55:52.549559 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.549386    2560 scope.go:117] "RemoveContainer" containerID="1b2c25b80bcf8eef3fffeeeb712bdd6618cdb3e86e526b13db190ff048c4ec6f"
Sep 11 17:55:52.558633 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.558608    2560 scope.go:117] "RemoveContainer" containerID="1b2c25b80bcf8eef3fffeeeb712bdd6618cdb3e86e526b13db190ff048c4ec6f"
Sep 11 17:55:52.558969 ip-10-0-138-148 kubenswrapper[2560]: E0911 17:55:52.558949    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b2c25b80bcf8eef3fffeeeb712bdd6618cdb3e86e526b13db190ff048c4ec6f\": container with ID starting with 1b2c25b80bcf8eef3fffeeeb712bdd6618cdb3e86e526b13db190ff048c4ec6f not found: ID does not exist" containerID="1b2c25b80bcf8eef3fffeeeb712bdd6618cdb3e86e526b13db190ff048c4ec6f"
Sep 11 17:55:52.559023 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.558987    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2c25b80bcf8eef3fffeeeb712bdd6618cdb3e86e526b13db190ff048c4ec6f"} err="failed to get container status \"1b2c25b80bcf8eef3fffeeeb712bdd6618cdb3e86e526b13db190ff048c4ec6f\": rpc error: code = NotFound desc = could not find container \"1b2c25b80bcf8eef3fffeeeb712bdd6618cdb3e86e526b13db190ff048c4ec6f\": container with ID starting with 1b2c25b80bcf8eef3fffeeeb712bdd6618cdb3e86e526b13db190ff048c4ec6f not found: ID does not exist"
Sep 11 17:55:52.575690 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.575659    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"]
Sep 11 17:55:52.582121 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:52.582089    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e145a56e-kserve-65b8b76455-xvt27"]
Sep 11 17:55:54.053434 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:55:54.053402    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0b0a00d-305a-4580-b243-948cab628790" path="/var/lib/kubelet/pods/e0b0a00d-305a-4580-b243-948cab628790/volumes"
Sep 11 17:56:26.023950 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:26.023861    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64d78656b9-72zss_b0975010-1a10-4076-814e-64ce01d14694/authorino/0.log"
Sep 11 17:56:30.672075 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:30.672037    2560 scope.go:117] "RemoveContainer" containerID="98248d965963af7758e8ccb2a2dbfcb109a3cd31ed2833bff319b9f89f4e3957"
Sep 11 17:56:30.680712 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:30.680691    2560 scope.go:117] "RemoveContainer" containerID="625919b60466b34524455e7cccfed1181e8b393e5c71b7853423dd993fc9c5d9"
Sep 11 17:56:32.425946 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:32.425901    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-d5vf9_e71d59dc-fde9-4176-afa5-8d6f015d6157/manager/0.log"
Sep 11 17:56:32.673434 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:32.673404    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f9c58fc78-jbpht_5db69f6a-8a40-4538-9770-278a8df11739/copy-manifests/0.log"
Sep 11 17:56:32.691864 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:32.691768    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f9c58fc78-jbpht_5db69f6a-8a40-4538-9770-278a8df11739/manager/0.log"
Sep 11 17:56:34.475931 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:34.475898    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64d78656b9-72zss_b0975010-1a10-4076-814e-64ce01d14694/authorino/0.log"
Sep 11 17:56:35.564264 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:35.564229    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg_1f502bdb-2db6-47fb-a8a8-5b53d96519eb/istio-proxy/0.log"
Sep 11 17:56:35.921316 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:35.921284    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5897fcd6c7-qgbz2_c9c38b6d-7101-4132-ae6e-377a97881082/kube-auth-proxy/0.log"
Sep 11 17:56:36.043017 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:36.042980    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wmvtc_24d8555f-802e-4533-8e21-d1eb611ba4a6/istio-proxy/1.log"
Sep 11 17:56:36.407338 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:36.407297    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8648dc8856-csvbr_594faf95-252b-4fef-ab68-b4ace396cdd7/router/0.log"
Sep 11 17:56:36.991002 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:36.990976    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-jbj8q_100017ae-b96c-4b9b-9e7c-b66f3e2ef228/main/0.log"
Sep 11 17:56:37.345194 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:37.345115    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-rmfvj_92d4801d-b38f-43e2-b2f7-d996ebbbf7fe/main/0.log"
Sep 11 17:56:37.469219 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:37.469191    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-pdgs6_b594845d-6db1-42db-a77c-dfbcd6b0db0f/main/0.log"
Sep 11 17:56:43.363499 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:43.363438    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:56:44.433838 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:44.433796    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log" limit=1024
Sep 11 17:56:46.516549 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:46.516514    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b8pb5_04be58fc-8937-4595-afcb-ab03d51fd1e7/global-pull-secret-syncer/0.log"
Sep 11 17:56:46.629360 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:46.629327    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5dl9t_bc62979d-5f6d-45f6-b2a2-34cd2228a46e/konnectivity-agent/0.log"
Sep 11 17:56:46.751959 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:46.751923    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-148.ec2.internal_fcffbb3555202a1c58f0c3380c4a7532/haproxy/0.log"
Sep 11 17:56:51.751397 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:51.751361    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64d78656b9-72zss_b0975010-1a10-4076-814e-64ce01d14694/authorino/0.log"
Sep 11 17:56:53.573859 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:53.573831    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76883eb1-c32d-47d4-9832-5f59dc8fcdfe/alertmanager/0.log"
Sep 11 17:56:53.604444 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:53.604411    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76883eb1-c32d-47d4-9832-5f59dc8fcdfe/config-reloader/0.log"
Sep 11 17:56:53.645077 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:53.645047    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76883eb1-c32d-47d4-9832-5f59dc8fcdfe/kube-rbac-proxy-web/0.log"
Sep 11 17:56:53.668681 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:53.668640    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76883eb1-c32d-47d4-9832-5f59dc8fcdfe/kube-rbac-proxy/0.log"
Sep 11 17:56:53.695789 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:53.695751    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76883eb1-c32d-47d4-9832-5f59dc8fcdfe/kube-rbac-proxy-metric/0.log"
Sep 11 17:56:53.719682 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:53.719656    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76883eb1-c32d-47d4-9832-5f59dc8fcdfe/prom-label-proxy/0.log"
Sep 11 17:56:53.743276 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:53.743240    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_76883eb1-c32d-47d4-9832-5f59dc8fcdfe/init-config-reloader/0.log"
Sep 11 17:56:53.793770 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:53.793732    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-j4htb_85d1d971-8409-4d04-a9c0-578ab1e8102a/cluster-monitoring-operator/0.log"
Sep 11 17:56:53.832284 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:53.832187    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-k4jzc_2ece35c6-9894-448b-9e3c-c90566b9d95e/kube-state-metrics/0.log"
Sep 11 17:56:53.861311 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:53.861272    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-k4jzc_2ece35c6-9894-448b-9e3c-c90566b9d95e/kube-rbac-proxy-main/0.log"
Sep 11 17:56:53.886303 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:53.886269    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-k4jzc_2ece35c6-9894-448b-9e3c-c90566b9d95e/kube-rbac-proxy-self/0.log"
Sep 11 17:56:53.914741 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:53.914710    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-57c896d4fd-9hc4p_76dde87e-bb30-4d65-9885-9fed9409ba76/metrics-server/0.log"
Sep 11 17:56:53.940119 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:53.940092    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-5jfwt_c8336388-5be6-4a25-8a3c-110e377e84da/monitoring-plugin/0.log"
Sep 11 17:56:54.148820 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:54.148788    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rfbv6_4b337361-c434-4668-9ad2-a527b1789840/node-exporter/0.log"
Sep 11 17:56:54.184114 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:54.184087    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rfbv6_4b337361-c434-4668-9ad2-a527b1789840/kube-rbac-proxy/0.log"
Sep 11 17:56:54.216488 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:54.216449    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rfbv6_4b337361-c434-4668-9ad2-a527b1789840/init-textfile/0.log"
Sep 11 17:56:54.619491 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:54.619359    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f64d99d5b-tqhfc_c909851f-bc7f-4cba-a450-9a2fadba8e71/telemeter-client/0.log"
Sep 11 17:56:54.642668 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:54.642633    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f64d99d5b-tqhfc_c909851f-bc7f-4cba-a450-9a2fadba8e71/reload/0.log"
Sep 11 17:56:54.664444 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:54.664412    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f64d99d5b-tqhfc_c909851f-bc7f-4cba-a450-9a2fadba8e71/kube-rbac-proxy/0.log"
Sep 11 17:56:56.012787 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.012758    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-7xhfb_ea721b9a-6947-44eb-b898-b331c3ad7d64/networking-console-plugin/0.log"
Sep 11 17:56:56.248992 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.248949    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"]
Sep 11 17:56:56.249333 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.249321    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e21c2f-4429-46d1-aff8-b58e6cbbb24d" containerName="main"
Sep 11 17:56:56.249333 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.249335    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e21c2f-4429-46d1-aff8-b58e6cbbb24d" containerName="main"
Sep 11 17:56:56.249424 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.249357    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0b0a00d-305a-4580-b243-948cab628790" containerName="main"
Sep 11 17:56:56.249424 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.249364    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0b0a00d-305a-4580-b243-948cab628790" containerName="main"
Sep 11 17:56:56.249538 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.249424    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0b0a00d-305a-4580-b243-948cab628790" containerName="main"
Sep 11 17:56:56.249538 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.249434    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e21c2f-4429-46d1-aff8-b58e6cbbb24d" containerName="main"
Sep 11 17:56:56.252766 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.252744    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.255215 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.255194    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbzkp\"/\"openshift-service-ca.crt\""
Sep 11 17:56:56.255215 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.255206    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbzkp\"/\"kube-root-ca.crt\""
Sep 11 17:56:56.256372 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.256352    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rbzkp\"/\"default-dockercfg-h9zpw\""
Sep 11 17:56:56.262251 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.262227    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"]
Sep 11 17:56:56.300664 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.300572    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4d2cf94-50c7-4ee0-bb42-77ce66376ada-sys\") pod \"perf-node-gather-daemonset-j9xcd\" (UID: \"b4d2cf94-50c7-4ee0-bb42-77ce66376ada\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.300664 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.300622    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b4d2cf94-50c7-4ee0-bb42-77ce66376ada-podres\") pod \"perf-node-gather-daemonset-j9xcd\" (UID: \"b4d2cf94-50c7-4ee0-bb42-77ce66376ada\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.300894 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.300737    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4d2cf94-50c7-4ee0-bb42-77ce66376ada-lib-modules\") pod \"perf-node-gather-daemonset-j9xcd\" (UID: \"b4d2cf94-50c7-4ee0-bb42-77ce66376ada\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.300894 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.300815    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b4d2cf94-50c7-4ee0-bb42-77ce66376ada-proc\") pod \"perf-node-gather-daemonset-j9xcd\" (UID: \"b4d2cf94-50c7-4ee0-bb42-77ce66376ada\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.300894 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.300850    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg8xd\" (UniqueName: \"kubernetes.io/projected/b4d2cf94-50c7-4ee0-bb42-77ce66376ada-kube-api-access-fg8xd\") pod \"perf-node-gather-daemonset-j9xcd\" (UID: \"b4d2cf94-50c7-4ee0-bb42-77ce66376ada\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.401742 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.401707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4d2cf94-50c7-4ee0-bb42-77ce66376ada-sys\") pod \"perf-node-gather-daemonset-j9xcd\" (UID: \"b4d2cf94-50c7-4ee0-bb42-77ce66376ada\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.401919 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.401755    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b4d2cf94-50c7-4ee0-bb42-77ce66376ada-podres\") pod \"perf-node-gather-daemonset-j9xcd\" (UID: \"b4d2cf94-50c7-4ee0-bb42-77ce66376ada\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.401919 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.401794    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4d2cf94-50c7-4ee0-bb42-77ce66376ada-lib-modules\") pod \"perf-node-gather-daemonset-j9xcd\" (UID: \"b4d2cf94-50c7-4ee0-bb42-77ce66376ada\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.401919 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.401795    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4d2cf94-50c7-4ee0-bb42-77ce66376ada-sys\") pod \"perf-node-gather-daemonset-j9xcd\" (UID: \"b4d2cf94-50c7-4ee0-bb42-77ce66376ada\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.401919 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.401868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b4d2cf94-50c7-4ee0-bb42-77ce66376ada-proc\") pod \"perf-node-gather-daemonset-j9xcd\" (UID: \"b4d2cf94-50c7-4ee0-bb42-77ce66376ada\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.401919 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.401915    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4d2cf94-50c7-4ee0-bb42-77ce66376ada-lib-modules\") pod \"perf-node-gather-daemonset-j9xcd\" (UID: \"b4d2cf94-50c7-4ee0-bb42-77ce66376ada\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.402096 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.401920    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg8xd\" (UniqueName: \"kubernetes.io/projected/b4d2cf94-50c7-4ee0-bb42-77ce66376ada-kube-api-access-fg8xd\") pod \"perf-node-gather-daemonset-j9xcd\" (UID: \"b4d2cf94-50c7-4ee0-bb42-77ce66376ada\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.402096 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.401933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b4d2cf94-50c7-4ee0-bb42-77ce66376ada-podres\") pod \"perf-node-gather-daemonset-j9xcd\" (UID: \"b4d2cf94-50c7-4ee0-bb42-77ce66376ada\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.402096 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.401978    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b4d2cf94-50c7-4ee0-bb42-77ce66376ada-proc\") pod \"perf-node-gather-daemonset-j9xcd\" (UID: \"b4d2cf94-50c7-4ee0-bb42-77ce66376ada\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.411851 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.411825    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg8xd\" (UniqueName: \"kubernetes.io/projected/b4d2cf94-50c7-4ee0-bb42-77ce66376ada-kube-api-access-fg8xd\") pod \"perf-node-gather-daemonset-j9xcd\" (UID: \"b4d2cf94-50c7-4ee0-bb42-77ce66376ada\") " pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.564380 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.564286    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:56.837537 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.837507    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"]
Sep 11 17:56:56.839686 ip-10-0-138-148 kubenswrapper[2560]: W0911 17:56:56.839648    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb4d2cf94_50c7_4ee0_bb42_77ce66376ada.slice/crio-a72a3eea4ff4e7c54acf15afdd711744e7003ba34db1c79db40263493d4cf19f WatchSource:0}: Error finding container a72a3eea4ff4e7c54acf15afdd711744e7003ba34db1c79db40263493d4cf19f: Status 404 returned error can't find the container with id a72a3eea4ff4e7c54acf15afdd711744e7003ba34db1c79db40263493d4cf19f
Sep 11 17:56:56.867667 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:56.867626    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd" event={"ID":"b4d2cf94-50c7-4ee0-bb42-77ce66376ada","Type":"ContainerStarted","Data":"a72a3eea4ff4e7c54acf15afdd711744e7003ba34db1c79db40263493d4cf19f"}
Sep 11 17:56:57.272152 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:57.272098    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-rbnrs_54ac5c72-1b5d-4813-a4c7-6f6f108752a2/download-server/0.log"
Sep 11 17:56:57.874275 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:57.874236    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd" event={"ID":"b4d2cf94-50c7-4ee0-bb42-77ce66376ada","Type":"ContainerStarted","Data":"6fce1e9df04f26767d556826502312aa4a91d9482a52e63bbd2d5ecb5855ae35"}
Sep 11 17:56:57.874524 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:57.874356    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:56:57.890626 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:57.890572    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd" podStartSLOduration=1.890553244 podStartE2EDuration="1.890553244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:56:57.890193079 +0000 UTC m=+2188.322645857" watchObservedRunningTime="2026-09-11 17:56:57.890553244 +0000 UTC m=+2188.323006021"
Sep 11 17:56:57.933295 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:57.933246    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-z9sdw_fcb7feab-00d8-4799-9647-43725d1cc29f/volume-data-source-validator/0.log"
Sep 11 17:56:59.170906 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:59.170871    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x2zzk_63bddd62-4518-490e-863d-f2d592e79ec3/dns/0.log"
Sep 11 17:56:59.198656 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:59.198610    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x2zzk_63bddd62-4518-490e-863d-f2d592e79ec3/kube-rbac-proxy/0.log"
Sep 11 17:56:59.333231 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:59.333203    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fmkwp_8ccc7a17-e4f2-48ac-a2b1-308eda0170f4/dns-node-resolver/0.log"
Sep 11 17:56:59.925000 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:56:59.924962    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-86565b78d5-ngh5h_5f1440e1-fa51-49ec-9ec8-686d89bcc606/registry/0.log"
Sep 11 17:57:00.004724 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:00.004694    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ttqn4_9ae8cc59-4697-44e8-9b49-3e97f11bdbb9/node-ca/0.log"
Sep 11 17:57:00.985538 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:00.985506    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg_1f502bdb-2db6-47fb-a8a8-5b53d96519eb/istio-proxy/0.log"
Sep 11 17:57:01.344924 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:01.344835    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5897fcd6c7-qgbz2_c9c38b6d-7101-4132-ae6e-377a97881082/kube-auth-proxy/0.log"
Sep 11 17:57:01.377615 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:01.377587    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wmvtc_24d8555f-802e-4533-8e21-d1eb611ba4a6/istio-proxy/0.log"
Sep 11 17:57:01.393108 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:01.393072    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wmvtc_24d8555f-802e-4533-8e21-d1eb611ba4a6/istio-proxy/1.log"
Sep 11 17:57:01.497905 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:01.497872    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8648dc8856-csvbr_594faf95-252b-4fef-ab68-b4ace396cdd7/router/0.log"
Sep 11 17:57:02.441150 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:02.441120    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zkwcj_2afe3f73-882c-41a4-b26b-a4f3cb0e4264/serve-healthcheck-canary/0.log"
Sep 11 17:57:02.913460 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:02.913423    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-24qlf_e353cc34-4593-4fa9-bba7-6509ecef06cc/insights-operator/0.log"
Sep 11 17:57:02.915621 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:02.915597    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-24qlf_e353cc34-4593-4fa9-bba7-6509ecef06cc/insights-operator/1.log"
Sep 11 17:57:03.020247 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:03.020217    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c5qhl_ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9/kube-rbac-proxy/0.log"
Sep 11 17:57:03.045568 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:03.045539    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c5qhl_ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9/exporter/0.log"
Sep 11 17:57:03.069321 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:03.069294    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c5qhl_ef3cab1a-7e90-4ee0-b19c-c1541f17f0e9/extractor/0.log"
Sep 11 17:57:03.889250 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:03.889217    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rbzkp/perf-node-gather-daemonset-j9xcd"
Sep 11 17:57:05.896264 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:05.896223    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-d5vf9_e71d59dc-fde9-4176-afa5-8d6f015d6157/manager/0.log"
Sep 11 17:57:06.068143 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:06.068052    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f9c58fc78-jbpht_5db69f6a-8a40-4538-9770-278a8df11739/manager/0.log"
Sep 11 17:57:06.087356 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:06.087329    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f9c58fc78-jbpht_5db69f6a-8a40-4538-9770-278a8df11739/copy-manifests/0.log"
Sep 11 17:57:07.291543 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:07.291513    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg_1f502bdb-2db6-47fb-a8a8-5b53d96519eb/istio-proxy/0.log"
Sep 11 17:57:07.424751 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:07.424720    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5786cf49b6-z49b8_8f5f8c43-f99c-41af-b9a0-8ebebf360ab6/manager/0.log"
Sep 11 17:57:07.609176 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:07.609144    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5897fcd6c7-qgbz2_c9c38b6d-7101-4132-ae6e-377a97881082/kube-auth-proxy/0.log"
Sep 11 17:57:07.632223 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:07.632195    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wmvtc_24d8555f-802e-4533-8e21-d1eb611ba4a6/istio-proxy/0.log"
Sep 11 17:57:07.648376 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:07.648344    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wmvtc_24d8555f-802e-4533-8e21-d1eb611ba4a6/istio-proxy/1.log"
Sep 11 17:57:07.721245 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:07.721208    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8648dc8856-csvbr_594faf95-252b-4fef-ab68-b4ace396cdd7/router/0.log"
Sep 11 17:57:09.130055 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:09.130024    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-dbjtg_1f502bdb-2db6-47fb-a8a8-5b53d96519eb/istio-proxy/0.log"
Sep 11 17:57:09.457670 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:09.457581    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5897fcd6c7-qgbz2_c9c38b6d-7101-4132-ae6e-377a97881082/kube-auth-proxy/0.log"
Sep 11 17:57:09.483546 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:09.483506    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wmvtc_24d8555f-802e-4533-8e21-d1eb611ba4a6/istio-proxy/0.log"
Sep 11 17:57:09.499897 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:09.499859    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wmvtc_24d8555f-802e-4533-8e21-d1eb611ba4a6/istio-proxy/1.log"
Sep 11 17:57:09.579943 ip-10-0-138-148 kubenswrapper[2560]: I0911 17:57:09.579904    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8648dc8856-csvbr_594faf95-252b-4fef-ab68-b4ace396cdd7/router/0.log"