Sep 17 23:04:24.179499 ip-10-0-140-44 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 23:04:24.179511 ip-10-0-140-44 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 23:04:24.179520 ip-10-0-140-44 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 23:04:24.179719 ip-10-0-140-44 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 23:04:34.428458 ip-10-0-140-44 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 23:04:34.428476 ip-10-0-140-44 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4dc5b7eed39a447a8bc6152978540877 --
Sep 17 23:06:40.298185 ip-10-0-140-44 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 23:06:43.772520 ip-10-0-140-44 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:06:43.772520 ip-10-0-140-44 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 23:06:43.772520 ip-10-0-140-44 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:06:43.772520 ip-10-0-140-44 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 23:06:43.772520 ip-10-0-140-44 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:06:43.773426 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.772613    2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 23:06:43.776390 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776373    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:06:43.776390 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776389    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776393    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776396    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776401    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776406    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776410    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776413    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776415    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776418    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776421    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776424    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776426    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776429    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776432    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776434    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776437    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776440    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776442    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776445    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:06:43.776454 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776449    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776453    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776456    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776460    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776463    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776468    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776472    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776475    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776478    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776480    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776483    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776488    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776491    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776494    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776496    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776499    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776502    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776505    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776509    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776512    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:06:43.776931 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776515    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776518    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776521    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776524    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776527    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776529    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776533    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776535    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776538    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776540    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776543    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776546    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776548    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776551    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776554    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776557    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776560    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776562    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776565    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776567    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:06:43.777456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776570    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776572    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776575    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776578    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776581    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776583    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776586    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776589    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776592    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776595    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776598    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776600    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776605    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776607    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776610    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776613    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776615    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776618    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776621    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776623    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:06:43.777967 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776626    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776628    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776631    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776634    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776636    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776639    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776642    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776645    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776648    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776650    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776653    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776656    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776659    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776662    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776664    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776667    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776670    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776672    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.776675    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777370    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:06:43.778488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777377    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777380    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777383    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777386    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777389    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777391    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777394    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777397    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777399    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777402    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777405    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777407    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777411    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777414    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777416    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777419    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777422    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777425    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777429    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777431    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:06:43.778999 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777434    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777445    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777448    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777450    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777453    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777456    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777458    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777461    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777464    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777467    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777470    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777473    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777476    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777479    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777482    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777484    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777487    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777490    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777492    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777495    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:06:43.779491 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777498    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777500    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777503    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777505    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777508    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777511    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777514    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777516    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777519    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777522    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777526    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777528    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777531    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777534    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777536    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777539    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777542    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777544    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777547    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777550    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:06:43.780015 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777552    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777555    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777557    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777562    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777565    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777568    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777571    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777574    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777576    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777579    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777582    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777584    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777587    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777593    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777596    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777599    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777602    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777604    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777608    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:06:43.780514 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777610    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777613    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777616    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777619    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777622    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777624    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777627    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777630    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777632    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777636    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777639    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777642    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777645    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777647    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777650    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777652    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777655    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777658    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.777660    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:06:43.781004 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778050    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778055    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778058    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778061    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778064    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778067    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778070    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778073    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778075    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778079    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778082    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778084    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778087    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778090    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778094    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778099    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778102    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778105    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778108    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:06:43.781486 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778111    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778114    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778116    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778119    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778121    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778124    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778127    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778129    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778133    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778136    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778138    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778141    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778143    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778146    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778148    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778151    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778153    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778159    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778161    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778164    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:06:43.782031 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778167    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778170    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778172    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778175    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778177    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778179    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778182    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778185    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778188    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778191    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778194    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778196    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778199    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778201    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778204    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778206    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778209    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778211    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778214    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778216    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:06:43.782556 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778219    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778222    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778225    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778227    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778230    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778232    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778235    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778237    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778239    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778244    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778246    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778249    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778253    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778256    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778259    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778262    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778264    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778267    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778270    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778273    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:06:43.783099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778277    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778281    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778284    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778286    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778289    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778292    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778294    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778296    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778299    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778302    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778304    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778307    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778309    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778312    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778314    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778317    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778319    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778322    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778324    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.778327    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:06:43.783586 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778404    2568 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778443    2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778449    2568 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778454    2568 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778458    2568 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778461    2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778465    2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778469    2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778473    2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778476    2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778479    2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778483    2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778488    2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778491    2568 flags.go:64] FLAG: --cgroup-root=""
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778495    2568 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778498    2568 flags.go:64] FLAG: --client-ca-file=""
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778501    2568 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778504    2568 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778509    2568 flags.go:64] FLAG: --cluster-domain=""
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778512    2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778515    2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778518    2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778522    2568 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778526    2568 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 23:06:43.784105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778529    2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778532    2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778535    2568 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778539    2568 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778541    2568 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778544    2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778548    2568 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778551    2568 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778557    2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778562    2568 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778565    2568 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778567    2568 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778570    2568 flags.go:64] FLAG: --enable-server="true"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778573    2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778578    2568 flags.go:64] FLAG: --event-burst="100"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778581    2568 flags.go:64] FLAG: --event-qps="50"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778584    2568 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778587    2568 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778590    2568 flags.go:64] FLAG: --eviction-hard=""
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778593    2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778597    2568 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778601    2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778604    2568 flags.go:64] FLAG: --eviction-soft=""
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778607    2568 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778610    2568 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 23:06:43.784678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778613    2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778616    2568 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778619    2568 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778622    2568 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778625    2568 flags.go:64] FLAG: --feature-gates=""
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778628    2568 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778631    2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778634    2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778637    2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778640    2568 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778643    2568 flags.go:64] FLAG: --help="false"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778646    2568 flags.go:64] FLAG: --hostname-override="ip-10-0-140-44.ec2.internal"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778649    2568 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778652    2568 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778655    2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778658    2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778663    2568 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778666    2568 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778668    2568 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778671    2568 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778674    2568 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778677    2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778681    2568 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778683    2568 flags.go:64] FLAG: --kube-reserved=""
Sep 17 23:06:43.785312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778686    2568 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778689    2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778692    2568 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778696    2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778699    2568 flags.go:64] FLAG: --lock-file=""
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778702    2568 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778705    2568 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778708    2568 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778714    2568 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778716    2568 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778719    2568 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778722    2568 flags.go:64] FLAG: --logging-format="text"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778725    2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778728    2568 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778731    2568 flags.go:64] FLAG: --manifest-url=""
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778734    2568 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778739    2568 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778741    2568 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778745    2568 flags.go:64] FLAG: --max-pods="110"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778748    2568 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778751    2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778754    2568 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778757    2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778760    2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778764    2568 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 23:06:43.785976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778767    2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778774    2568 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778777    2568 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778780    2568 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778783    2568 flags.go:64] FLAG: --pod-cidr=""
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778786    2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778791    2568 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778794    2568 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778798    2568 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778801    2568 flags.go:64] FLAG: --port="10250"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778803    2568 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778808    2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0315cfb547753869a"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778811    2568 flags.go:64] FLAG: --qos-reserved=""
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778814    2568 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778818    2568 flags.go:64] FLAG: --register-node="true"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778820    2568 flags.go:64] FLAG: --register-with-taints=""
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778824    2568 flags.go:64] FLAG: --registry-burst="10"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778827    2568 flags.go:64] FLAG: --registry-qps="5"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778830    2568 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778833    2568 flags.go:64] FLAG: --reserved-memory=""
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778837    2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778840    2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778843    2568 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778846    2568 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778848    2568 flags.go:64] FLAG: --runonce="false"
Sep 17 23:06:43.786598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778851    2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778854    2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778857    2568 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778860    2568 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778875    2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778879    2568 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778883    2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778886    2568 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778889    2568 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778892    2568 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778895    2568 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778898    2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778901    2568 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778904    2568 flags.go:64] FLAG: --system-cgroups=""
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778907    2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778912    2568 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778915    2568 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778918    2568 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778924    2568 flags.go:64] FLAG: --tls-min-version=""
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778926    2568 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778929    2568 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778933    2568 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778936    2568 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778939    2568 flags.go:64] FLAG: --v="2"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778943    2568 flags.go:64] FLAG: --version="false"
Sep 17 23:06:43.787229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778947    2568 flags.go:64] FLAG: --vmodule=""
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778950    2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.778953    2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779066    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779071    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779074    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779077    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779080    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779082    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779085    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779088    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779091    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779093    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779098    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779101    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779103    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779106    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779109    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779112    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779115    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:06:43.787873 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779117    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779120    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779123    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779125    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779128    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779132    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779134    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779137    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779140    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779143    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779146    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779149    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779152    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779155    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779157    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779160    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779163    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779165    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779168    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779170    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:06:43.788396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779174    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779176    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779179    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779181    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779185    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779190    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779193    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779197    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779200    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779202    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779205    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779208    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779210    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779213    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779216    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779218    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779221    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779226    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779229    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:06:43.788885 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779233    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779237    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779241    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779245    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779249    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779253    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779256    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779260    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779263    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779265    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779268    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779271    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779274    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779277    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779280    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779283    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779287    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779291    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779295    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:06:43.789360 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779298    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779301    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779304    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779307    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779310    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779313    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779316    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779319    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779322    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779325    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779328    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779332    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779335    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779338    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779341    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779344    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779347    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779351    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779353    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779356    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:06:43.789828 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779359    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:06:43.790343 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779361    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:06:43.790343 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779364    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:06:43.790343 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.779366    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:06:43.790343 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.780144    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:06:43.790343 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.787084    2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 23:06:43.790343 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.787098    2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 23:06:43.790343 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.787118    2568 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 23:06:43.790343 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.787127    2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 23:06:43.790343 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787181    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:06:43.790343 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787186    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:06:43.790343 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787190    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:06:43.790343 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787193    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:06:43.790343 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787196    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:06:43.790343 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787199    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:06:43.790343 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787203    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787206    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787209    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787212    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787215    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787218    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787221    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787224    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787227    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787230    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787233    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787236    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787240    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787242    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787246    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787249    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787252    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787255    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787258    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787261    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:06:43.790726 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787263    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787266    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787269    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787272    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787275    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787278    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787281    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787284    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787286    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787289    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787291    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787294    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787297    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787300    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787302    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787305    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787307    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787310    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787313    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787315    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:06:43.791240 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787318    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787321    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787323    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787326    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787328    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787331    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787333    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787336    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787339    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787342    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787344    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787347    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787349    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787352    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787354    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787358    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787361    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787363    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787366    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787369    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:06:43.791746 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787372    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787375    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787380    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787383    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787388    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787391    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787394    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787397    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787400    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787402    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787405    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787408    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787410    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787413    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787416    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787418    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787421    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787424    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787428    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:06:43.792237 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787432    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:06:43.792708 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787435    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:06:43.792708 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787439    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:06:43.792708 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787442    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:06:43.792708 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787444    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:06:43.792708 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787447    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:06:43.792708 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787450    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:06:43.792708 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787455    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:06:43.792708 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787458    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:06:43.792708 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787461    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:06:43.792708 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787464    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:06:43.792708 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787467    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:06:43.792708 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787470    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:06:43.792708 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787472    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:06:43.792708 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.787477    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:06:43.792708 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787586    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787590    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787593    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787596    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787599    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787602    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787604    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787607    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787609    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787612    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787615    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787618    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787621    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787623    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787626    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787629    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787631    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787642    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787645    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787648    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:06:43.793099 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787651    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787654    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787658    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787661    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787663    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787666    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787669    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787671    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787674    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787676    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787679    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787681    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787684    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787686    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787689    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787691    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787694    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787696    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787699    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787702    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:06:43.793591 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787704    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787707    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787709    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787712    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787715    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787717    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787720    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787722    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787725    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787727    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787730    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787733    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787736    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787739    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787742    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787744    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787747    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787749    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787752    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:06:43.794214 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787754    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787758    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787761    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787764    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787767    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787770    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787772    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787775    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787778    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787781    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787783    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787786    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787788    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787791    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787793    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787796    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787798    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787802    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787806    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:06:43.794678 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787809    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787812    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787815    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787818    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787822    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787825    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787829    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787832    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787835    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787837    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787840    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787843    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787845    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787848    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787850    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787853    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787856    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787858    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787861    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787876    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:06:43.795447 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:43.787880    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:06:43.796244 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.787886    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:06:43.796244 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.787987    2568 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 23:06:43.796244 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.790016    2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 23:06:43.796244 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.791296    2568 server.go:1019] "Starting client certificate rotation"
Sep 17 23:06:43.796244 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.791401    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:06:43.796244 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.791949    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:06:43.796244 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.793767    2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:06:43.797791 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.797768    2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:06:43.801992 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.801973    2568 log.go:25] "Validated CRI v1 runtime API"
Sep 17 23:06:43.804629 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.804615    2568 log.go:25] "Validated CRI v1 image API"
Sep 17 23:06:43.805354 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.805341    2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 23:06:43.807903 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.807885    2568 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7ca6f463-852a-45b0-bb6a-9aedcfa9b766:/dev/nvme0n1p3 ab343710-69df-4302-871c-0437503b7969:/dev/nvme0n1p4]
Sep 17 23:06:43.807944 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.807904    2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 23:06:43.812708 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.812597    2568 manager.go:216] Machine: {Timestamp:2026-09-17 23:06:43.812002417 +0000 UTC m=+2.035678157 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3084511 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21827c25355c0426f709b87516d5c9 SystemUUID:ec21827c-2535-5c04-26f7-09b87516d5c9 BootID:4dc5b7ee-d39a-447a-8bc6-152978540877 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cd:8c:95:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cd:8c:95:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:db:f0:c3:a5:45 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 23:06:43.812708 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.812704    2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 23:06:43.812830 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.812818    2568 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 23:06:43.814344 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.814317    2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 23:06:43.814476 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.814348    2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-44.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 23:06:43.816212 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.816197    2568 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 23:06:43.816212 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.816214    2568 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 23:06:43.816340 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.816222    2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:06:43.816340 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.816233    2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:06:43.816340 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.816301    2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 23:06:43.820742 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.820728    2568 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:06:43.820878 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.820854    2568 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 23:06:43.820918 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.820903    2568 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 23:06:43.820946 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.820917    2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 23:06:43.820946 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.820941    2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 23:06:43.820996 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.820954    2568 kubelet.go:398] "Adding apiserver pod source"
Sep 17 23:06:43.820996 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.820966    2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 23:06:43.821380 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.821369    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:06:43.821443 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.821388    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:06:43.823658 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.823630    2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 23:06:43.830006 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.829986    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:06:43.830684 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.830669    2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 23:06:43.830732 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.830700    2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 23:06:43.831766 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.831749    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 23:06:43.831766 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.831768    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 23:06:43.831766 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.831775    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 23:06:43.831766 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.831780    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 23:06:43.831979 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.831790    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 23:06:43.831979 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.831795    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 23:06:43.831979 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.831809    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 23:06:43.831979 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.831814    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 23:06:43.831979 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.831821    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 23:06:43.831979 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.831827    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 23:06:43.831979 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.831836    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 23:06:43.831979 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.831841    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 23:06:43.831979 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.831879    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 23:06:43.831979 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.831888    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 23:06:43.833486 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:43.833461    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-44.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 23:06:43.833486 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:43.833467    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 23:06:43.835802 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.835789    2568 server.go:1268] "Started kubelet"
Sep 17 23:06:43.835942 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.835918    2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 23:06:43.836100 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.835862    2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 23:06:43.836100 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.836089    2568 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 23:06:43.836666 ip-10-0-140-44 systemd[1]: Started Kubernetes Kubelet.
Sep 17 23:06:43.837238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.837221    2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 23:06:43.837301 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.837239    2568 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 23:06:43.841859 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:43.841840    2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 23:06:43.842789 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.842775    2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-44.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 23:06:43.843981 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.843968    2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 23:06:43.844041 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.843992    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 23:06:43.844088 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:43.844040    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:43.844591 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.844568    2568 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 23:06:43.844591 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.844587    2568 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 23:06:43.844709 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.844691    2568 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 23:06:43.844709 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.844703    2568 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 23:06:43.844834 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.844576    2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 23:06:43.844834 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.844824    2568 factory.go:55] Registering systemd factory
Sep 17 23:06:43.844997 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.844849    2568 factory.go:223] Registration of the systemd container factory successfully
Sep 17 23:06:43.845105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.845083    2568 factory.go:153] Registering CRI-O factory
Sep 17 23:06:43.845155 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.845108    2568 factory.go:223] Registration of the crio container factory successfully
Sep 17 23:06:43.845196 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.845156    2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 23:06:43.845196 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.845172    2568 factory.go:103] Registering Raw factory
Sep 17 23:06:43.845196 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.845182    2568 manager.go:1196] Started watching for new ooms in manager
Sep 17 23:06:43.846096 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.846082    2568 manager.go:349] Starting recovery of all containers
Sep 17 23:06:43.850444 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.850425    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z2brb"
Sep 17 23:06:43.864377 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:43.864349    2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-44.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 23:06:43.864563 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:43.864534    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 23:06:43.864678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.864651    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z2brb"
Sep 17 23:06:43.868280 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.868263    2568 manager.go:354] Recovery completed
Sep 17 23:06:43.872537 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.872523    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:06:43.874327 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.874309    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:06:43.874395 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.874340    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:06:43.874395 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.874351    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:06:43.874951 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.874936    2568 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 23:06:43.874951 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.874947    2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 23:06:43.874951 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.874961    2568 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:06:43.875109 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.874974    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:06:43.875669 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.875656    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:06:43.875732 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.875680    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:06:43.875732 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.875689    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:06:43.876989 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.876975    2568 policy_none.go:49] "None policy: Start"
Sep 17 23:06:43.877030 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.876996    2568 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 23:06:43.877030 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.877007    2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 23:06:43.878884 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.878857    2568 policy_none.go:47] "Start"
Sep 17 23:06:43.919663 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.919488    2568 manager.go:337] "Starting Device Plugin manager"
Sep 17 23:06:43.937094 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:43.919714    2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 23:06:43.937094 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.919728    2568 server.go:87] "Starting device plugin registration server"
Sep 17 23:06:43.937094 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.919981    2568 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 23:06:43.937094 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.920001    2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 23:06:43.937094 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.920097    2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 23:06:43.937094 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.920180    2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 23:06:43.937094 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.920189    2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 23:06:43.937094 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:43.920707    2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 23:06:43.937094 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:43.920759    2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:43.982078 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.982039    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 23:06:43.983264 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.983245    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 23:06:43.983341 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.983314    2568 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 23:06:43.983391 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.983342    2568 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 23:06:43.983391 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:43.983388    2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 23:06:43.985387 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:43.985358    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:06:44.020302 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.020269    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:06:44.022301 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.022285    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:06:44.022374 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.022317    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:06:44.022374 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.022332    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:06:44.022374 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.022356    2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.028678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.028622    2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.028678 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:44.028646    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-44.ec2.internal\": node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:44.045015 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:44.044987    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:44.083499 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.083465    2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-44.ec2.internal"]
Sep 17 23:06:44.083661 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.083542    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:06:44.084473 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.084456    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:06:44.084569 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.084484    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:06:44.084569 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.084493    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:06:44.086746 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.086733    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:06:44.086860 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.086844    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.086916 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.086892    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:06:44.087467 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.087453    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:06:44.087505 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.087480    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:06:44.087505 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.087493    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:06:44.087573 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.087458    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:06:44.087573 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.087544    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:06:44.087573 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.087553    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:06:44.089683 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.089668    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.089776 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.089691    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:06:44.090368 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.090349    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:06:44.090470 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.090379    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:06:44.090470 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.090390    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:06:44.116880 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:44.116836    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-44.ec2.internal\" not found" node="ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.120409 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:44.120391    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-44.ec2.internal\" not found" node="ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.145789 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:44.145759    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:44.146733 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.146716    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/551ba4d2dbc3a651cf174938293b65bf-config\") pod \"kube-apiserver-proxy-ip-10-0-140-44.ec2.internal\" (UID: \"551ba4d2dbc3a651cf174938293b65bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.146820 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.146745    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f92a2085ca93766d4ef040134671a4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal\" (UID: \"7f92a2085ca93766d4ef040134671a4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.146820 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.146770    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f92a2085ca93766d4ef040134671a4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal\" (UID: \"7f92a2085ca93766d4ef040134671a4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.246116 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:44.246084    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:44.247214 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.247196    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f92a2085ca93766d4ef040134671a4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal\" (UID: \"7f92a2085ca93766d4ef040134671a4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.247265 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.247232    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/551ba4d2dbc3a651cf174938293b65bf-config\") pod \"kube-apiserver-proxy-ip-10-0-140-44.ec2.internal\" (UID: \"551ba4d2dbc3a651cf174938293b65bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.247265 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.247252    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f92a2085ca93766d4ef040134671a4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal\" (UID: \"7f92a2085ca93766d4ef040134671a4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.247340 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.247279    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/551ba4d2dbc3a651cf174938293b65bf-config\") pod \"kube-apiserver-proxy-ip-10-0-140-44.ec2.internal\" (UID: \"551ba4d2dbc3a651cf174938293b65bf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.247340 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.247287    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f92a2085ca93766d4ef040134671a4c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal\" (UID: \"7f92a2085ca93766d4ef040134671a4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.247340 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.247321    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f92a2085ca93766d4ef040134671a4c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal\" (UID: \"7f92a2085ca93766d4ef040134671a4c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.346920 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:44.346834    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:44.420059 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.420025    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.423763 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.423717    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-44.ec2.internal"
Sep 17 23:06:44.447859 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:44.447827    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:44.548301 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:44.548264    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:44.648844 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:44.648785    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:44.749243 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:44.749209    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:44.789491 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.789466    2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:06:44.790774 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.790761    2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 23:06:44.790896 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.790878    2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:06:44.850129 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:44.850100    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:44.871004 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.870968    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 23:01:43 +0000 UTC" deadline="2028-06-30 11:41:50.261350238 +0000 UTC"
Sep 17 23:06:44.871004 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:44.870996    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15636h35m5.390357935s"
Sep 17 23:06:44.950625 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:44.950545    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:45.050934 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:45.050907    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:45.151383 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:45.151349    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:45.251909 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:45.251860    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:45.254547 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:45.254514    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f92a2085ca93766d4ef040134671a4c.slice/crio-14cde6f2b11c333b8d151c5733a05036e805fb422f58c3c485e3b73d88be3297 WatchSource:0}: Error finding container 14cde6f2b11c333b8d151c5733a05036e805fb422f58c3c485e3b73d88be3297: Status 404 returned error can't find the container with id 14cde6f2b11c333b8d151c5733a05036e805fb422f58c3c485e3b73d88be3297
Sep 17 23:06:45.254823 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:45.254805    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551ba4d2dbc3a651cf174938293b65bf.slice/crio-78e763c777f155895415a39b9f5cd4ed081ad98911253a9befb05f19ed930aa8 WatchSource:0}: Error finding container 78e763c777f155895415a39b9f5cd4ed081ad98911253a9befb05f19ed930aa8: Status 404 returned error can't find the container with id 78e763c777f155895415a39b9f5cd4ed081ad98911253a9befb05f19ed930aa8
Sep 17 23:06:45.260353 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.260338    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:06:45.352917 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:45.352858    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-44.ec2.internal\" not found"
Sep 17 23:06:45.384168 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.384145    2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:06:45.406391 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.406364    2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:06:45.445144 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.445113    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal"
Sep 17 23:06:45.456951 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.456934    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:06:45.458257 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.458244    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-44.ec2.internal"
Sep 17 23:06:45.557401 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.557379    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:06:45.658953 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.658928    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:06:45.822965 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.822890    2568 apiserver.go:52] "Watching apiserver"
Sep 17 23:06:45.830358 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.830328    2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 23:06:45.830741 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.830711    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-9vxwv","openshift-network-operator/iptables-alerter-xzrtv","openshift-ovn-kubernetes/ovnkube-node-zlpxj","kube-system/global-pull-secret-syncer-rw5hp","kube-system/konnectivity-agent-5xj9x","kube-system/kube-apiserver-proxy-ip-10-0-140-44.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6","openshift-dns/node-resolver-qkccv","openshift-image-registry/node-ca-dghj8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal","openshift-multus/multus-additional-cni-plugins-9sjsw","openshift-cluster-node-tuning-operator/tuned-rjcdn","openshift-multus/multus-xg9wq","openshift-multus/network-metrics-daemon-r5mz6"]
Sep 17 23:06:45.833535 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.833505    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.835992 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.835970    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.836471 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.836294    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 23:06:45.836471 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.836335    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hskzb\""
Sep 17 23:06:45.836471 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.836382    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 23:06:45.836732 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.836502    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 23:06:45.838528 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.838142    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 23:06:45.838528 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.838289    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.838528 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.838454    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 23:06:45.838890 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.838858    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9wxxw\""
Sep 17 23:06:45.838974 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.838861    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 23:06:45.839260 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.839239    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 23:06:45.840570 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.840550    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 23:06:45.840823 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.840804    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:45.840965 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:45.840888    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:06:45.841237 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.841186    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 23:06:45.841525 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.841375    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 23:06:45.841816 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.841643    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wcj72\""
Sep 17 23:06:45.841816 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.841672    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 23:06:45.842540 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.842289    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 23:06:45.842540 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.842490    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 23:06:45.843474 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.843150    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5xj9x"
Sep 17 23:06:45.845152 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.845084    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 23:06:45.845597 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.845577    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:45.845703 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:45.845635    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:06:45.846148 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.846004    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 23:06:45.846148 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.846068    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jhjbj\""
Sep 17 23:06:45.846148 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.846131    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 23:06:45.849603 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.848362    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dghj8"
Sep 17 23:06:45.851772 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.850759    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 23:06:45.851772 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.851174    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:06:45.851772 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:45.851253    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:06:45.851982 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.851847    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h67vq\""
Sep 17 23:06:45.852133 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.852117    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 23:06:45.852459 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.852239    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 23:06:45.853459 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.853443    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xzrtv"
Sep 17 23:06:45.855815 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.855623    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-cnibin\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.855815 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.855663    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gckvt\" (UniqueName: \"kubernetes.io/projected/3560e9bb-709c-4dfc-9407-85406aec3f3e-kube-api-access-gckvt\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.855815 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.855689    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-systemd-units\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.855815 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.855712    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-run-systemd\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.855815 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.855734    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-etc-openvswitch\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.855815 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.855755    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-log-socket\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.855815 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.855778    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/13416a44-5328-4fa3-9d5e-83a2ab0ad655-env-overrides\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.856217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.855818    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/13416a44-5328-4fa3-9d5e-83a2ab0ad655-ovnkube-script-lib\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.856217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.855858    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-socket-dir\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.856217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.855894    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:06:45.856217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.855910    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-run-k8s-cni-cncf-io\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.856217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.855937    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnqzj\" (UniqueName: \"kubernetes.io/projected/13416a44-5328-4fa3-9d5e-83a2ab0ad655-kube-api-access-lnqzj\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.856217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.855961    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qkccv"
Sep 17 23:06:45.856217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856016    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/67f097ec-5a12-42f3-84bf-a66968a7a323-konnectivity-ca\") pod \"konnectivity-agent-5xj9x\" (UID: \"67f097ec-5a12-42f3-84bf-a66968a7a323\") " pod="kube-system/konnectivity-agent-5xj9x"
Sep 17 23:06:45.856217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856043    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgtcx\" (UniqueName: \"kubernetes.io/projected/76c81e49-6a97-4454-945f-9fd9bea45db2-kube-api-access-mgtcx\") pod \"network-metrics-daemon-r5mz6\" (UID: \"76c81e49-6a97-4454-945f-9fd9bea45db2\") " pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:45.856217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856068    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-registration-dir\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.856217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856091    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-etc-selinux\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.856217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856150    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-var-lib-kubelet\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.856217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856175    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-cni-bin\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.856217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856202    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856242    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856269    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-etc-kubernetes\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856295    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856320    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp844\" (UniqueName: \"kubernetes.io/projected/f70c2877-9d27-4ac0-8453-85018507bdef-kube-api-access-xp844\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856344    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-system-cni-dir\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856402    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-run-netns\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856426    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-slash\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856451    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/13416a44-5328-4fa3-9d5e-83a2ab0ad655-ovn-node-metrics-cert\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856490    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-device-dir\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856514    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-var-lib-cni-bin\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856541    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-run-netns\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856518    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856580    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-node-log\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856614    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-run-ovn-kubernetes\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856643    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/75be2122-acf1-43e5-902d-c1d574d42792-kubelet-config\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856674    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/75be2122-acf1-43e5-902d-c1d574d42792-dbus\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:45.856894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856699    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/67f097ec-5a12-42f3-84bf-a66968a7a323-agent-certs\") pod \"konnectivity-agent-5xj9x\" (UID: \"67f097ec-5a12-42f3-84bf-a66968a7a323\") " pod="kube-system/konnectivity-agent-5xj9x"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856727    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3560e9bb-709c-4dfc-9407-85406aec3f3e-cni-binary-copy\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856751    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-multus-socket-dir-parent\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856773    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3560e9bb-709c-4dfc-9407-85406aec3f3e-multus-daemon-config\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856796    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-kubelet\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856828    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs\") pod \"network-metrics-daemon-r5mz6\" (UID: \"76c81e49-6a97-4454-945f-9fd9bea45db2\") " pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856882    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-sys-fs\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856905    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-hostroot\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856927    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-multus-conf-dir\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.856948    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-run-multus-certs\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.857016    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-var-lib-openvswitch\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.857047    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-cni-netd\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.857070    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-multus-cni-dir\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.857094    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-var-lib-cni-multus\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.857105    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lcv7t\""
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.857117    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-run-openvswitch\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.857140    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-run-ovn\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.857687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.857164    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/13416a44-5328-4fa3-9d5e-83a2ab0ad655-ovnkube-config\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.858482 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.857208    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:45.858482 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.858358    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:45.858753 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.858642    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-frpq6\""
Sep 17 23:06:45.859051 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.858881    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 23:06:45.859051 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.858976    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 23:06:45.860743 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.860714    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:45.861300 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.861056    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ppcd7\""
Sep 17 23:06:45.861300 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.861166    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 23:06:45.861300 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.861213    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 23:06:45.862210 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.862188    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:06:45.863361 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.863338    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 23:06:45.863967 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.863777    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:06:45.864046 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.863974    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k9brr\""
Sep 17 23:06:45.883953 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.883930    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4plrw"
Sep 17 23:06:45.889415 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.889387    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4plrw"
Sep 17 23:06:45.946142 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.946068    2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.957779    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-modprobe-d\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.957815    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa-hosts-file\") pod \"node-resolver-qkccv\" (UID: \"2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa\") " pod="openshift-dns/node-resolver-qkccv"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.957852    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-var-lib-kubelet\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.957975    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958001    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vncd8\" (UniqueName: \"kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8\") pod \"network-check-target-9vxwv\" (UID: \"9e17f611-03ca-4632-938e-9af066f3bc1c\") " pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958023    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13483042-c3be-4ef8-b615-72852899eade-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958048    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/67f097ec-5a12-42f3-84bf-a66968a7a323-konnectivity-ca\") pod \"konnectivity-agent-5xj9x\" (UID: \"67f097ec-5a12-42f3-84bf-a66968a7a323\") " pod="kube-system/konnectivity-agent-5xj9x"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958077    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-etc-kubernetes\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958098    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-run-netns\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958119    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/75be2122-acf1-43e5-902d-c1d574d42792-dbus\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958140    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13483042-c3be-4ef8-b615-72852899eade-cnibin\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958162    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13483042-c3be-4ef8-b615-72852899eade-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958183    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-run-netns\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958212    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/75be2122-acf1-43e5-902d-c1d574d42792-kubelet-config\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958233    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13483042-c3be-4ef8-b615-72852899eade-cni-binary-copy\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:45.958856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958256    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13483042-c3be-4ef8-b615-72852899eade-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958307    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-var-lib-kubelet\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958340    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f537520e-e943-4837-a871-3142a15f6fba-serviceca\") pod \"node-ca-dghj8\" (UID: \"f537520e-e943-4837-a871-3142a15f6fba\") " pod="openshift-image-registry/node-ca-dghj8"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958343    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958398    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-kubelet\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958419    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs\") pod \"network-metrics-daemon-r5mz6\" (UID: \"76c81e49-6a97-4454-945f-9fd9bea45db2\") " pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958447    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdtfc\" (UniqueName: \"kubernetes.io/projected/8be07c34-232d-4784-a1b4-0172240ebf7d-kube-api-access-xdtfc\") pod \"iptables-alerter-xzrtv\" (UID: \"8be07c34-232d-4784-a1b4-0172240ebf7d\") " pod="openshift-network-operator/iptables-alerter-xzrtv"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958508    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-hostroot\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958529    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/13416a44-5328-4fa3-9d5e-83a2ab0ad655-ovnkube-config\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958558    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958573    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/75be2122-acf1-43e5-902d-c1d574d42792-dbus\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958582    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-sys\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958605    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-etc-kubernetes\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958628    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-kubelet\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958655    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-run-netns\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958669    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-run-netns\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:45.958693    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:45.958693    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958708    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-hostroot\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.959775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958712    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-lib-modules\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958734    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-host\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958770    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-run-openvswitch\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:45.958811    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs podName:76c81e49-6a97-4454-945f-9fd9bea45db2 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:46.458769396 +0000 UTC m=+4.682445123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs") pod "network-metrics-daemon-r5mz6" (UID: "76c81e49-6a97-4454-945f-9fd9bea45db2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958810    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-run-openvswitch\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:45.958854    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret podName:75be2122-acf1-43e5-902d-c1d574d42792 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:46.458836229 +0000 UTC m=+4.682511960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret") pod "global-pull-secret-syncer-rw5hp" (UID: "75be2122-acf1-43e5-902d-c1d574d42792") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958892    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d35f966-2cf2-46cc-9069-e971900f7430-etc-tuned\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958919    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f537520e-e943-4837-a871-3142a15f6fba-host\") pod \"node-ca-dghj8\" (UID: \"f537520e-e943-4837-a871-3142a15f6fba\") " pod="openshift-image-registry/node-ca-dghj8"
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958943    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa-tmp-dir\") pod \"node-resolver-qkccv\" (UID: \"2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa\") " pod="openshift-dns/node-resolver-qkccv"
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958949    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/75be2122-acf1-43e5-902d-c1d574d42792-kubelet-config\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958955    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/67f097ec-5a12-42f3-84bf-a66968a7a323-konnectivity-ca\") pod \"konnectivity-agent-5xj9x\" (UID: \"67f097ec-5a12-42f3-84bf-a66968a7a323\") " pod="kube-system/konnectivity-agent-5xj9x"
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.958981    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-cnibin\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959010    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gckvt\" (UniqueName: \"kubernetes.io/projected/3560e9bb-709c-4dfc-9407-85406aec3f3e-kube-api-access-gckvt\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959035    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-systemd-units\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959056    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-etc-openvswitch\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959077    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-log-socket\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.960797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959101    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-var-lib-kubelet\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959117    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-cnibin\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959126    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-socket-dir\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959148    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mdhx\" (UniqueName: \"kubernetes.io/projected/6d35f966-2cf2-46cc-9069-e971900f7430-kube-api-access-9mdhx\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959157    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-log-socket\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959159    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/13416a44-5328-4fa3-9d5e-83a2ab0ad655-ovnkube-config\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959172    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-registration-dir\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959114    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-systemd-units\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959208    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-etc-openvswitch\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959220    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-registration-dir\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959258    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-etc-selinux\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959294    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-cni-bin\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959319    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-socket-dir\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959350    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-cni-bin\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959393    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959398    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-etc-selinux\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959436    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp844\" (UniqueName: \"kubernetes.io/projected/f70c2877-9d27-4ac0-8453-85018507bdef-kube-api-access-xp844\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.961549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959469    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-system-cni-dir\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959438    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959520    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-slash\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959546    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/13416a44-5328-4fa3-9d5e-83a2ab0ad655-ovn-node-metrics-cert\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959571    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-systemd\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959595    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-device-dir\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959600    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-slash\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959617    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-var-lib-cni-bin\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959545    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-system-cni-dir\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959648    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-node-log\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959673    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-run-ovn-kubernetes\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959681    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-device-dir\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959693    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-var-lib-cni-bin\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959735    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-node-log\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959740    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-run-ovn-kubernetes\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959696    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/67f097ec-5a12-42f3-84bf-a66968a7a323-agent-certs\") pod \"konnectivity-agent-5xj9x\" (UID: \"67f097ec-5a12-42f3-84bf-a66968a7a323\") " pod="kube-system/konnectivity-agent-5xj9x"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959780    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-sysctl-d\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:45.962238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959803    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q858\" (UniqueName: \"kubernetes.io/projected/2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa-kube-api-access-8q858\") pod \"node-resolver-qkccv\" (UID: \"2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa\") " pod="openshift-dns/node-resolver-qkccv"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959830    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3560e9bb-709c-4dfc-9407-85406aec3f3e-cni-binary-copy\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959854    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-multus-socket-dir-parent\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959860    2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959895    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3560e9bb-709c-4dfc-9407-85406aec3f3e-multus-daemon-config\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959919    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-sysconfig\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959941    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-run\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959965    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-sys-fs\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.959988    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-multus-conf-dir\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960011    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-run-multus-certs\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960033    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-var-lib-openvswitch\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960057    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-cni-netd\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960080    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8be07c34-232d-4784-a1b4-0172240ebf7d-iptables-alerter-script\") pod \"iptables-alerter-xzrtv\" (UID: \"8be07c34-232d-4784-a1b4-0172240ebf7d\") " pod="openshift-network-operator/iptables-alerter-xzrtv"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960103    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d35f966-2cf2-46cc-9069-e971900f7430-tmp\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960127    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-multus-cni-dir\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960152    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-var-lib-cni-multus\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960182    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-run-ovn\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960205    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f70c2877-9d27-4ac0-8453-85018507bdef-sys-fs\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.963079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960207    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13483042-c3be-4ef8-b615-72852899eade-system-cni-dir\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960251    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-run-systemd\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960315    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-run-ovn\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960319    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-host-cni-netd\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960348    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/13416a44-5328-4fa3-9d5e-83a2ab0ad655-env-overrides\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960354    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-run-systemd\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960374    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/13416a44-5328-4fa3-9d5e-83a2ab0ad655-ovnkube-script-lib\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960388    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-multus-conf-dir\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960398    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnqzj\" (UniqueName: \"kubernetes.io/projected/13416a44-5328-4fa3-9d5e-83a2ab0ad655-kube-api-access-lnqzj\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960426    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8d8s\" (UniqueName: \"kubernetes.io/projected/13483042-c3be-4ef8-b615-72852899eade-kube-api-access-v8d8s\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960442    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-multus-cni-dir\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960452    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-kubernetes\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960476    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/13416a44-5328-4fa3-9d5e-83a2ab0ad655-var-lib-openvswitch\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960480    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-sysctl-conf\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960509    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-var-lib-cni-multus\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960505    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkp7t\" (UniqueName: \"kubernetes.io/projected/f537520e-e943-4837-a871-3142a15f6fba-kube-api-access-nkp7t\") pod \"node-ca-dghj8\" (UID: \"f537520e-e943-4837-a871-3142a15f6fba\") " pod="openshift-image-registry/node-ca-dghj8"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960579    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-run-multus-certs\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.963940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960579    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-run-k8s-cni-cncf-io\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.964802 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960611    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-host-run-k8s-cni-cncf-io\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.964802 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960616    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgtcx\" (UniqueName: \"kubernetes.io/projected/76c81e49-6a97-4454-945f-9fd9bea45db2-kube-api-access-mgtcx\") pod \"network-metrics-daemon-r5mz6\" (UID: \"76c81e49-6a97-4454-945f-9fd9bea45db2\") " pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:45.964802 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960644    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8be07c34-232d-4784-a1b4-0172240ebf7d-host-slash\") pod \"iptables-alerter-xzrtv\" (UID: \"8be07c34-232d-4784-a1b4-0172240ebf7d\") " pod="openshift-network-operator/iptables-alerter-xzrtv"
Sep 17 23:06:45.964802 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.960856    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3560e9bb-709c-4dfc-9407-85406aec3f3e-multus-daemon-config\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.964802 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.961046    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3560e9bb-709c-4dfc-9407-85406aec3f3e-multus-socket-dir-parent\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.964802 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.961111    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/13416a44-5328-4fa3-9d5e-83a2ab0ad655-ovnkube-script-lib\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.964802 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.961399    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3560e9bb-709c-4dfc-9407-85406aec3f3e-cni-binary-copy\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.964802 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.961412    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/13416a44-5328-4fa3-9d5e-83a2ab0ad655-env-overrides\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.965445 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.965386    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/13416a44-5328-4fa3-9d5e-83a2ab0ad655-ovn-node-metrics-cert\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.965551 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.965509    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/67f097ec-5a12-42f3-84bf-a66968a7a323-agent-certs\") pod \"konnectivity-agent-5xj9x\" (UID: \"67f097ec-5a12-42f3-84bf-a66968a7a323\") " pod="kube-system/konnectivity-agent-5xj9x"
Sep 17 23:06:45.968613 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.968283    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gckvt\" (UniqueName: \"kubernetes.io/projected/3560e9bb-709c-4dfc-9407-85406aec3f3e-kube-api-access-gckvt\") pod \"multus-xg9wq\" (UID: \"3560e9bb-709c-4dfc-9407-85406aec3f3e\") " pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:45.973104 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.973057    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp844\" (UniqueName: \"kubernetes.io/projected/f70c2877-9d27-4ac0-8453-85018507bdef-kube-api-access-xp844\") pod \"aws-ebs-csi-driver-node-5gdc6\" (UID: \"f70c2877-9d27-4ac0-8453-85018507bdef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:45.973571 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.973525    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnqzj\" (UniqueName: \"kubernetes.io/projected/13416a44-5328-4fa3-9d5e-83a2ab0ad655-kube-api-access-lnqzj\") pod \"ovnkube-node-zlpxj\" (UID: \"13416a44-5328-4fa3-9d5e-83a2ab0ad655\") " pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:45.974121 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.974090    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgtcx\" (UniqueName: \"kubernetes.io/projected/76c81e49-6a97-4454-945f-9fd9bea45db2-kube-api-access-mgtcx\") pod \"network-metrics-daemon-r5mz6\" (UID: \"76c81e49-6a97-4454-945f-9fd9bea45db2\") " pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:45.990266 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.990183    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal" event={"ID":"7f92a2085ca93766d4ef040134671a4c","Type":"ContainerStarted","Data":"14cde6f2b11c333b8d151c5733a05036e805fb422f58c3c485e3b73d88be3297"}
Sep 17 23:06:45.991954 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:45.991904    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-44.ec2.internal" event={"ID":"551ba4d2dbc3a651cf174938293b65bf","Type":"ContainerStarted","Data":"78e763c777f155895415a39b9f5cd4ed081ad98911253a9befb05f19ed930aa8"}
Sep 17 23:06:46.061211 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061173    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8be07c34-232d-4784-a1b4-0172240ebf7d-iptables-alerter-script\") pod \"iptables-alerter-xzrtv\" (UID: \"8be07c34-232d-4784-a1b4-0172240ebf7d\") " pod="openshift-network-operator/iptables-alerter-xzrtv"
Sep 17 23:06:46.061211 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061219    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d35f966-2cf2-46cc-9069-e971900f7430-tmp\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.061464 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061249    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13483042-c3be-4ef8-b615-72852899eade-system-cni-dir\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:46.061464 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061320    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8d8s\" (UniqueName: \"kubernetes.io/projected/13483042-c3be-4ef8-b615-72852899eade-kube-api-access-v8d8s\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:46.061464 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061347    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-kubernetes\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.061464 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061371    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-sysctl-conf\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.061464 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061395    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkp7t\" (UniqueName: \"kubernetes.io/projected/f537520e-e943-4837-a871-3142a15f6fba-kube-api-access-nkp7t\") pod \"node-ca-dghj8\" (UID: \"f537520e-e943-4837-a871-3142a15f6fba\") " pod="openshift-image-registry/node-ca-dghj8"
Sep 17 23:06:46.061464 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061420    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8be07c34-232d-4784-a1b4-0172240ebf7d-host-slash\") pod \"iptables-alerter-xzrtv\" (UID: \"8be07c34-232d-4784-a1b4-0172240ebf7d\") " pod="openshift-network-operator/iptables-alerter-xzrtv"
Sep 17 23:06:46.061464 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061443    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-modprobe-d\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.061464 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061466    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa-hosts-file\") pod \"node-resolver-qkccv\" (UID: \"2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa\") " pod="openshift-dns/node-resolver-qkccv"
Sep 17 23:06:46.061840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061494    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncd8\" (UniqueName: \"kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8\") pod \"network-check-target-9vxwv\" (UID: \"9e17f611-03ca-4632-938e-9af066f3bc1c\") " pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:06:46.061840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061521    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13483042-c3be-4ef8-b615-72852899eade-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:46.061840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061547    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13483042-c3be-4ef8-b615-72852899eade-cnibin\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:46.061840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061572    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13483042-c3be-4ef8-b615-72852899eade-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:46.061840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061597    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13483042-c3be-4ef8-b615-72852899eade-cni-binary-copy\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:46.061840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061621    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13483042-c3be-4ef8-b615-72852899eade-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:46.061840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061643    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f537520e-e943-4837-a871-3142a15f6fba-serviceca\") pod \"node-ca-dghj8\" (UID: \"f537520e-e943-4837-a871-3142a15f6fba\") " pod="openshift-image-registry/node-ca-dghj8"
Sep 17 23:06:46.061840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061680    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdtfc\" (UniqueName: \"kubernetes.io/projected/8be07c34-232d-4784-a1b4-0172240ebf7d-kube-api-access-xdtfc\") pod \"iptables-alerter-xzrtv\" (UID: \"8be07c34-232d-4784-a1b4-0172240ebf7d\") " pod="openshift-network-operator/iptables-alerter-xzrtv"
Sep 17 23:06:46.061840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061718    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-sys\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.061840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061739    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-lib-modules\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.061840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061763    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-host\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.061840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061785    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d35f966-2cf2-46cc-9069-e971900f7430-etc-tuned\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.061840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061804    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f537520e-e943-4837-a871-3142a15f6fba-host\") pod \"node-ca-dghj8\" (UID: \"f537520e-e943-4837-a871-3142a15f6fba\") " pod="openshift-image-registry/node-ca-dghj8"
Sep 17 23:06:46.061840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061843    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa-tmp-dir\") pod \"node-resolver-qkccv\" (UID: \"2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa\") " pod="openshift-dns/node-resolver-qkccv"
Sep 17 23:06:46.062517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061852    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa-hosts-file\") pod \"node-resolver-qkccv\" (UID: \"2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa\") " pod="openshift-dns/node-resolver-qkccv"
Sep 17 23:06:46.062517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061896    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-var-lib-kubelet\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.062517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061915    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13483042-c3be-4ef8-b615-72852899eade-system-cni-dir\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:46.062517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061923    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mdhx\" (UniqueName: \"kubernetes.io/projected/6d35f966-2cf2-46cc-9069-e971900f7430-kube-api-access-9mdhx\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.062517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061968    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-systemd\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.062517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.061998    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-sysctl-d\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.062517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.062021    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q858\" (UniqueName: \"kubernetes.io/projected/2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa-kube-api-access-8q858\") pod \"node-resolver-qkccv\" (UID: \"2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa\") " pod="openshift-dns/node-resolver-qkccv"
Sep 17 23:06:46.062517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.062055    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-sysconfig\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.062517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.062080    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-run\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.062517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.062160    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-run\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.062517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.062235    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-sys\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.062517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.062239    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-kubernetes\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.062517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.062380    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-sysctl-conf\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.062517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.062506    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-lib-modules\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.063200 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.062558    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-host\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.063200 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.062575    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8be07c34-232d-4784-a1b4-0172240ebf7d-host-slash\") pod \"iptables-alerter-xzrtv\" (UID: \"8be07c34-232d-4784-a1b4-0172240ebf7d\") " pod="openshift-network-operator/iptables-alerter-xzrtv"
Sep 17 23:06:46.063200 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.062657    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-modprobe-d\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.063200 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.062702    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-systemd\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.063200 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.062779    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-sysctl-d\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.063200 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.063039    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-etc-sysconfig\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.063457 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.063257    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13483042-c3be-4ef8-b615-72852899eade-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:46.063457 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.063334    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13483042-c3be-4ef8-b615-72852899eade-cnibin\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:46.063561 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.063539    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13483042-c3be-4ef8-b615-72852899eade-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:46.063824 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.063728    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f537520e-e943-4837-a871-3142a15f6fba-host\") pod \"node-ca-dghj8\" (UID: \"f537520e-e943-4837-a871-3142a15f6fba\") " pod="openshift-image-registry/node-ca-dghj8"
Sep 17 23:06:46.064224 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.064040    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa-tmp-dir\") pod \"node-resolver-qkccv\" (UID: \"2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa\") " pod="openshift-dns/node-resolver-qkccv"
Sep 17 23:06:46.064224 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.064078    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13483042-c3be-4ef8-b615-72852899eade-cni-binary-copy\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:46.064224 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.064124    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d35f966-2cf2-46cc-9069-e971900f7430-var-lib-kubelet\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.064447 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.064316    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f537520e-e943-4837-a871-3142a15f6fba-serviceca\") pod \"node-ca-dghj8\" (UID: \"f537520e-e943-4837-a871-3142a15f6fba\") " pod="openshift-image-registry/node-ca-dghj8"
Sep 17 23:06:46.064622 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.064568    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13483042-c3be-4ef8-b615-72852899eade-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:46.064622 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.064602    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8be07c34-232d-4784-a1b4-0172240ebf7d-iptables-alerter-script\") pod \"iptables-alerter-xzrtv\" (UID: \"8be07c34-232d-4784-a1b4-0172240ebf7d\") " pod="openshift-network-operator/iptables-alerter-xzrtv"
Sep 17 23:06:46.065797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.065763    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d35f966-2cf2-46cc-9069-e971900f7430-tmp\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.067362 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.067340    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d35f966-2cf2-46cc-9069-e971900f7430-etc-tuned\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.068020 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:46.067745    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:46.068020 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:46.067773    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:46.068020 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:46.067787    2568 projected.go:196] Error preparing data for projected volume kube-api-access-vncd8 for pod openshift-network-diagnostics/network-check-target-9vxwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:46.068020 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:46.067852    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8 podName:9e17f611-03ca-4632-938e-9af066f3bc1c nodeName:}" failed. No retries permitted until 2026-09-17 23:06:46.567827417 +0000 UTC m=+4.791503147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vncd8" (UniqueName: "kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8") pod "network-check-target-9vxwv" (UID: "9e17f611-03ca-4632-938e-9af066f3bc1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:46.070658 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.070210    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mdhx\" (UniqueName: \"kubernetes.io/projected/6d35f966-2cf2-46cc-9069-e971900f7430-kube-api-access-9mdhx\") pod \"tuned-rjcdn\" (UID: \"6d35f966-2cf2-46cc-9069-e971900f7430\") " pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.070658 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.070614    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkp7t\" (UniqueName: \"kubernetes.io/projected/f537520e-e943-4837-a871-3142a15f6fba-kube-api-access-nkp7t\") pod \"node-ca-dghj8\" (UID: \"f537520e-e943-4837-a871-3142a15f6fba\") " pod="openshift-image-registry/node-ca-dghj8"
Sep 17 23:06:46.072251 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.072189    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdtfc\" (UniqueName: \"kubernetes.io/projected/8be07c34-232d-4784-a1b4-0172240ebf7d-kube-api-access-xdtfc\") pod \"iptables-alerter-xzrtv\" (UID: \"8be07c34-232d-4784-a1b4-0172240ebf7d\") " pod="openshift-network-operator/iptables-alerter-xzrtv"
Sep 17 23:06:46.072345 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.072270    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q858\" (UniqueName: \"kubernetes.io/projected/2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa-kube-api-access-8q858\") pod \"node-resolver-qkccv\" (UID: \"2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa\") " pod="openshift-dns/node-resolver-qkccv"
Sep 17 23:06:46.074014 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.073944    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8d8s\" (UniqueName: \"kubernetes.io/projected/13483042-c3be-4ef8-b615-72852899eade-kube-api-access-v8d8s\") pod \"multus-additional-cni-plugins-9sjsw\" (UID: \"13483042-c3be-4ef8-b615-72852899eade\") " pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:46.147968 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.147932    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6"
Sep 17 23:06:46.155397 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.155340    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xg9wq"
Sep 17 23:06:46.164833 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.164808    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:06:46.171653 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.171621    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5xj9x"
Sep 17 23:06:46.179281 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.179256    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dghj8"
Sep 17 23:06:46.188023 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.188002    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xzrtv"
Sep 17 23:06:46.194953 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.194928    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qkccv"
Sep 17 23:06:46.203639 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.203609    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9sjsw"
Sep 17 23:06:46.209437 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.209409    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rjcdn"
Sep 17 23:06:46.465004 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.464921    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs\") pod \"network-metrics-daemon-r5mz6\" (UID: \"76c81e49-6a97-4454-945f-9fd9bea45db2\") " pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:46.465004 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.464970    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:46.465218 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:46.465110    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:46.465218 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:46.465110    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:46.465218 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:46.465165    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret podName:75be2122-acf1-43e5-902d-c1d574d42792 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:47.465152022 +0000 UTC m=+5.688827748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret") pod "global-pull-secret-syncer-rw5hp" (UID: "75be2122-acf1-43e5-902d-c1d574d42792") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:46.465218 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:46.465178    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs podName:76c81e49-6a97-4454-945f-9fd9bea45db2 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:47.465172369 +0000 UTC m=+5.688848094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs") pod "network-metrics-daemon-r5mz6" (UID: "76c81e49-6a97-4454-945f-9fd9bea45db2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:46.666684 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.666640    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncd8\" (UniqueName: \"kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8\") pod \"network-check-target-9vxwv\" (UID: \"9e17f611-03ca-4632-938e-9af066f3bc1c\") " pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:06:46.666861 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:46.666821    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:46.666944 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:46.666879    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:46.666944 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:46.666897    2568 projected.go:196] Error preparing data for projected volume kube-api-access-vncd8 for pod openshift-network-diagnostics/network-check-target-9vxwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:46.667022 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:46.666965    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8 podName:9e17f611-03ca-4632-938e-9af066f3bc1c nodeName:}" failed. No retries permitted until 2026-09-17 23:06:47.666945699 +0000 UTC m=+5.890621446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vncd8" (UniqueName: "kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8") pod "network-check-target-9vxwv" (UID: "9e17f611-03ca-4632-938e-9af066f3bc1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:46.892575 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.892526    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:01:45 +0000 UTC" deadline="2028-05-25 10:43:55.863111738 +0000 UTC"
Sep 17 23:06:46.892575 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:46.892569    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14771h37m8.970546093s"
Sep 17 23:06:47.155102 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:47.154830    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70c2877_9d27_4ac0_8453_85018507bdef.slice/crio-f4f771abc3ffa9788f7c698e61566d0879faad4521ae57cf6a8f6c476c386690 WatchSource:0}: Error finding container f4f771abc3ffa9788f7c698e61566d0879faad4521ae57cf6a8f6c476c386690: Status 404 returned error can't find the container with id f4f771abc3ffa9788f7c698e61566d0879faad4521ae57cf6a8f6c476c386690
Sep 17 23:06:47.168860 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:47.168823    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf537520e_e943_4837_a871_3142a15f6fba.slice/crio-41fa09f60380cc3d842fbab8f4048a8bd17629720496b9c20757bc2347436260 WatchSource:0}: Error finding container 41fa09f60380cc3d842fbab8f4048a8bd17629720496b9c20757bc2347436260: Status 404 returned error can't find the container with id 41fa09f60380cc3d842fbab8f4048a8bd17629720496b9c20757bc2347436260
Sep 17 23:06:47.174632 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:47.174604    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d385719_4e1c_4eb3_8866_9fc9b3cdd6aa.slice/crio-cc567671ab9f55085e070a23bfcfdc0c8b2d8c79ab2ee9fd22a52a7cf1a1cad7 WatchSource:0}: Error finding container cc567671ab9f55085e070a23bfcfdc0c8b2d8c79ab2ee9fd22a52a7cf1a1cad7: Status 404 returned error can't find the container with id cc567671ab9f55085e070a23bfcfdc0c8b2d8c79ab2ee9fd22a52a7cf1a1cad7
Sep 17 23:06:47.175512 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:47.175474    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13483042_c3be_4ef8_b615_72852899eade.slice/crio-a48a9769a4dfa8dd16982d048280ab20329685f3dc717c7fc3dd1af5156a3885 WatchSource:0}: Error finding container a48a9769a4dfa8dd16982d048280ab20329685f3dc717c7fc3dd1af5156a3885: Status 404 returned error can't find the container with id a48a9769a4dfa8dd16982d048280ab20329685f3dc717c7fc3dd1af5156a3885
Sep 17 23:06:47.176456 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:47.176415    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d35f966_2cf2_46cc_9069_e971900f7430.slice/crio-40ea9db557690f3f13d0e02950a1a87b084051baaab28bf60e5748e68b1f62e5 WatchSource:0}: Error finding container 40ea9db557690f3f13d0e02950a1a87b084051baaab28bf60e5748e68b1f62e5: Status 404 returned error can't find the container with id 40ea9db557690f3f13d0e02950a1a87b084051baaab28bf60e5748e68b1f62e5
Sep 17 23:06:47.178769 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:47.178746    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f097ec_5a12_42f3_84bf_a66968a7a323.slice/crio-7aba04deca4e97e2bed0f778147e1dec0fe63da86a5d9f5a98d24f56650f4887 WatchSource:0}: Error finding container 7aba04deca4e97e2bed0f778147e1dec0fe63da86a5d9f5a98d24f56650f4887: Status 404 returned error can't find the container with id 7aba04deca4e97e2bed0f778147e1dec0fe63da86a5d9f5a98d24f56650f4887
Sep 17 23:06:47.180121 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:47.180102    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13416a44_5328_4fa3_9d5e_83a2ab0ad655.slice/crio-b8d807d92fadac55c0530d2d32ae550f55ddfea220a38ca3f3eda8ded9d9090c WatchSource:0}: Error finding container b8d807d92fadac55c0530d2d32ae550f55ddfea220a38ca3f3eda8ded9d9090c: Status 404 returned error can't find the container with id b8d807d92fadac55c0530d2d32ae550f55ddfea220a38ca3f3eda8ded9d9090c
Sep 17 23:06:47.203564 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:06:47.203539    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be07c34_232d_4784_a1b4_0172240ebf7d.slice/crio-05acbfd0bd79a7aa3547b57608db944aede5d7607a926d2cedc3ca245fa15308 WatchSource:0}: Error finding container 05acbfd0bd79a7aa3547b57608db944aede5d7607a926d2cedc3ca245fa15308: Status 404 returned error can't find the container with id 05acbfd0bd79a7aa3547b57608db944aede5d7607a926d2cedc3ca245fa15308
Sep 17 23:06:47.473279 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:47.473091    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs\") pod \"network-metrics-daemon-r5mz6\" (UID: \"76c81e49-6a97-4454-945f-9fd9bea45db2\") " pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:47.473425 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:47.473291    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:47.473425 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:47.473232    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:47.473425 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:47.473388    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:47.473425 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:47.473409    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs podName:76c81e49-6a97-4454-945f-9fd9bea45db2 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:49.473390507 +0000 UTC m=+7.697066239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs") pod "network-metrics-daemon-r5mz6" (UID: "76c81e49-6a97-4454-945f-9fd9bea45db2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:47.473646 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:47.473430    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret podName:75be2122-acf1-43e5-902d-c1d574d42792 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:49.473420063 +0000 UTC m=+7.697095790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret") pod "global-pull-secret-syncer-rw5hp" (UID: "75be2122-acf1-43e5-902d-c1d574d42792") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:47.679017 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:47.678094    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncd8\" (UniqueName: \"kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8\") pod \"network-check-target-9vxwv\" (UID: \"9e17f611-03ca-4632-938e-9af066f3bc1c\") " pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:06:47.679017 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:47.678325    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:47.679017 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:47.678351    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:47.679017 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:47.678364    2568 projected.go:196] Error preparing data for projected volume kube-api-access-vncd8 for pod openshift-network-diagnostics/network-check-target-9vxwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:47.679017 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:47.678424    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8 podName:9e17f611-03ca-4632-938e-9af066f3bc1c nodeName:}" failed. No retries permitted until 2026-09-17 23:06:49.678404841 +0000 UTC m=+7.902080577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vncd8" (UniqueName: "kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8") pod "network-check-target-9vxwv" (UID: "9e17f611-03ca-4632-938e-9af066f3bc1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:47.893428 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:47.893354    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:01:45 +0000 UTC" deadline="2028-06-27 16:08:40.165067856 +0000 UTC"
Sep 17 23:06:47.893428 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:47.893414    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15569h1m52.271658274s"
Sep 17 23:06:47.985056 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:47.984979    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:47.985222 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:47.985115    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:06:47.985557 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:47.985536    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:47.985673 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:47.985653    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:06:47.987770 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:47.987642    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:06:47.987770 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:47.987730    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:06:48.019222 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:48.018035    2568 generic.go:356] "Generic (PLEG): container finished" podID="7f92a2085ca93766d4ef040134671a4c" containerID="a0c1348ccd509a32fe4704537b3ca7510259225ddb7b6363ca0ef5acbd6e09e3" exitCode=0
Sep 17 23:06:48.019222 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:48.018128    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal" event={"ID":"7f92a2085ca93766d4ef040134671a4c","Type":"ContainerDied","Data":"a0c1348ccd509a32fe4704537b3ca7510259225ddb7b6363ca0ef5acbd6e09e3"}
Sep 17 23:06:48.034890 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:48.034000    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-44.ec2.internal" event={"ID":"551ba4d2dbc3a651cf174938293b65bf","Type":"ContainerStarted","Data":"a6026dfa0ef96edd21c50df5dcd4015bc02a7230fb7a3a324f9d7b3397d660ec"}
Sep 17 23:06:48.040611 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:48.040581    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xzrtv" event={"ID":"8be07c34-232d-4784-a1b4-0172240ebf7d","Type":"ContainerStarted","Data":"05acbfd0bd79a7aa3547b57608db944aede5d7607a926d2cedc3ca245fa15308"}
Sep 17 23:06:48.043934 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:48.043742    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj" event={"ID":"13416a44-5328-4fa3-9d5e-83a2ab0ad655","Type":"ContainerStarted","Data":"b8d807d92fadac55c0530d2d32ae550f55ddfea220a38ca3f3eda8ded9d9090c"}
Sep 17 23:06:48.046559 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:48.046498    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qkccv" event={"ID":"2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa","Type":"ContainerStarted","Data":"cc567671ab9f55085e070a23bfcfdc0c8b2d8c79ab2ee9fd22a52a7cf1a1cad7"}
Sep 17 23:06:48.060307 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:48.060228    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dghj8" event={"ID":"f537520e-e943-4837-a871-3142a15f6fba","Type":"ContainerStarted","Data":"41fa09f60380cc3d842fbab8f4048a8bd17629720496b9c20757bc2347436260"}
Sep 17 23:06:48.072835 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:48.072791    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6" event={"ID":"f70c2877-9d27-4ac0-8453-85018507bdef","Type":"ContainerStarted","Data":"f4f771abc3ffa9788f7c698e61566d0879faad4521ae57cf6a8f6c476c386690"}
Sep 17 23:06:48.076310 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:48.076276    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5xj9x" event={"ID":"67f097ec-5a12-42f3-84bf-a66968a7a323","Type":"ContainerStarted","Data":"7aba04deca4e97e2bed0f778147e1dec0fe63da86a5d9f5a98d24f56650f4887"}
Sep 17 23:06:48.083196 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:48.083126    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rjcdn" event={"ID":"6d35f966-2cf2-46cc-9069-e971900f7430","Type":"ContainerStarted","Data":"40ea9db557690f3f13d0e02950a1a87b084051baaab28bf60e5748e68b1f62e5"}
Sep 17 23:06:48.096660 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:48.096619    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sjsw" event={"ID":"13483042-c3be-4ef8-b615-72852899eade","Type":"ContainerStarted","Data":"a48a9769a4dfa8dd16982d048280ab20329685f3dc717c7fc3dd1af5156a3885"}
Sep 17 23:06:48.119126 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:48.119029    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xg9wq" event={"ID":"3560e9bb-709c-4dfc-9407-85406aec3f3e","Type":"ContainerStarted","Data":"56e63271757f4d271b9f68e0b64a60d6e607ee6504d6c7dae0e071ef56aa499f"}
Sep 17 23:06:49.138999 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:49.138935    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal" event={"ID":"7f92a2085ca93766d4ef040134671a4c","Type":"ContainerStarted","Data":"c360f10fd873bd2adc4feacf462d1437b76af64c848fdb33d99ea8c39ec0f1c2"}
Sep 17 23:06:49.159347 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:49.159273    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-44.ec2.internal" podStartSLOduration=4.159249739 podStartE2EDuration="4.159249739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:49.157982448 +0000 UTC m=+7.381658207" watchObservedRunningTime="2026-09-17 23:06:49.159249739 +0000 UTC m=+7.382925488"
Sep 17 23:06:49.160256 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:49.160121    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-44.ec2.internal" podStartSLOduration=4.160109079 podStartE2EDuration="4.160109079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:48.050826451 +0000 UTC m=+6.274502197" watchObservedRunningTime="2026-09-17 23:06:49.160109079 +0000 UTC m=+7.383784828"
Sep 17 23:06:49.495225 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:49.495118    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs\") pod \"network-metrics-daemon-r5mz6\" (UID: \"76c81e49-6a97-4454-945f-9fd9bea45db2\") " pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:49.495225 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:49.495172    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:49.495437 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:49.495310    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:49.495437 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:49.495372    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret podName:75be2122-acf1-43e5-902d-c1d574d42792 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:53.495354209 +0000 UTC m=+11.719029948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret") pod "global-pull-secret-syncer-rw5hp" (UID: "75be2122-acf1-43e5-902d-c1d574d42792") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:49.495826 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:49.495805    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:49.495949 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:49.495878    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs podName:76c81e49-6a97-4454-945f-9fd9bea45db2 nodeName:}" failed. No retries permitted until 2026-09-17 23:06:53.495847214 +0000 UTC m=+11.719522946 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs") pod "network-metrics-daemon-r5mz6" (UID: "76c81e49-6a97-4454-945f-9fd9bea45db2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:49.697169 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:49.697042    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncd8\" (UniqueName: \"kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8\") pod \"network-check-target-9vxwv\" (UID: \"9e17f611-03ca-4632-938e-9af066f3bc1c\") " pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:06:49.697358 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:49.697244    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:49.697358 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:49.697281    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:49.697358 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:49.697295    2568 projected.go:196] Error preparing data for projected volume kube-api-access-vncd8 for pod openshift-network-diagnostics/network-check-target-9vxwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:49.697358 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:49.697357    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8 podName:9e17f611-03ca-4632-938e-9af066f3bc1c nodeName:}" failed. No retries permitted until 2026-09-17 23:06:53.697337893 +0000 UTC m=+11.921013631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vncd8" (UniqueName: "kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8") pod "network-check-target-9vxwv" (UID: "9e17f611-03ca-4632-938e-9af066f3bc1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:49.984613 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:49.984347    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:49.984613 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:49.984487    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:06:49.984978 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:49.984885    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:06:49.985061 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:49.984985    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:06:49.985061 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:49.985041    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:49.985164 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:49.985105    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:06:51.989781 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:51.988911    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:51.989781 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:51.989055    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:06:51.989781 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:51.989498    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:06:51.989781 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:51.989609    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:06:51.989781 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:51.989665    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:51.989781 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:51.989740    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:06:53.527566 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:53.527525    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs\") pod \"network-metrics-daemon-r5mz6\" (UID: \"76c81e49-6a97-4454-945f-9fd9bea45db2\") " pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:53.528050 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:53.527579    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:53.528050 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:53.527714    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:53.528050 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:53.527786    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret podName:75be2122-acf1-43e5-902d-c1d574d42792 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:01.527766897 +0000 UTC m=+19.751442624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret") pod "global-pull-secret-syncer-rw5hp" (UID: "75be2122-acf1-43e5-902d-c1d574d42792") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:06:53.528050 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:53.527714    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:53.528050 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:53.527918    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs podName:76c81e49-6a97-4454-945f-9fd9bea45db2 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:01.52790342 +0000 UTC m=+19.751579170 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs") pod "network-metrics-daemon-r5mz6" (UID: "76c81e49-6a97-4454-945f-9fd9bea45db2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:06:53.729668 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:53.729445    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncd8\" (UniqueName: \"kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8\") pod \"network-check-target-9vxwv\" (UID: \"9e17f611-03ca-4632-938e-9af066f3bc1c\") " pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:06:53.729668 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:53.729631    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:06:53.729668 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:53.729651    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:06:53.729668 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:53.729664    2568 projected.go:196] Error preparing data for projected volume kube-api-access-vncd8 for pod openshift-network-diagnostics/network-check-target-9vxwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:53.730052 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:53.729728    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8 podName:9e17f611-03ca-4632-938e-9af066f3bc1c nodeName:}" failed. No retries permitted until 2026-09-17 23:07:01.729711044 +0000 UTC m=+19.953386772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vncd8" (UniqueName: "kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8") pod "network-check-target-9vxwv" (UID: "9e17f611-03ca-4632-938e-9af066f3bc1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:06:53.988162 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:53.987985    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:06:53.988162 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:53.988008    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:53.988162 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:53.988008    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:53.988162 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:53.988105    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:06:53.988470 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:53.988206    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:06:53.988654 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:53.988594    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:06:55.984700 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:55.984629    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:55.985193 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:55.984822    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:06:55.985261 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:55.985201    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:06:55.985261 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:55.985230    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:55.985364 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:55.985309    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:06:55.985418 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:55.985393    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:06:57.983888 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:57.983835    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:06:57.983888 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:57.983884    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:57.984402 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:57.983836    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:57.984402 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:57.983976    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:06:57.984402 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:57.984089    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:06:57.984402 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:57.984171    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:06:59.983722 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:59.983681    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:06:59.984262 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:59.983687    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:06:59.984262 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:59.983822    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:06:59.984262 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:06:59.983687    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:06:59.984262 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:59.983915    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:06:59.984262 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:06:59.983985    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:07:01.586395 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:01.586354    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs\") pod \"network-metrics-daemon-r5mz6\" (UID: \"76c81e49-6a97-4454-945f-9fd9bea45db2\") " pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:01.586395 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:01.586398    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:01.586910 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:01.586514    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:07:01.586910 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:01.586572    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret podName:75be2122-acf1-43e5-902d-c1d574d42792 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:17.586557513 +0000 UTC m=+35.810233239 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret") pod "global-pull-secret-syncer-rw5hp" (UID: "75be2122-acf1-43e5-902d-c1d574d42792") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:07:01.586910 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:01.586515    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:07:01.586910 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:01.586617    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs podName:76c81e49-6a97-4454-945f-9fd9bea45db2 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:17.586607219 +0000 UTC m=+35.810282946 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs") pod "network-metrics-daemon-r5mz6" (UID: "76c81e49-6a97-4454-945f-9fd9bea45db2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:07:01.787405 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:01.787364    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncd8\" (UniqueName: \"kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8\") pod \"network-check-target-9vxwv\" (UID: \"9e17f611-03ca-4632-938e-9af066f3bc1c\") " pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:01.787582 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:01.787557    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:07:01.787661 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:01.787585    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:07:01.787661 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:01.787599    2568 projected.go:196] Error preparing data for projected volume kube-api-access-vncd8 for pod openshift-network-diagnostics/network-check-target-9vxwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:07:01.787762 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:01.787668    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8 podName:9e17f611-03ca-4632-938e-9af066f3bc1c nodeName:}" failed. No retries permitted until 2026-09-17 23:07:17.787649136 +0000 UTC m=+36.011324876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vncd8" (UniqueName: "kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8") pod "network-check-target-9vxwv" (UID: "9e17f611-03ca-4632-938e-9af066f3bc1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:07:01.984344 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:01.984244    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:01.984344 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:01.984288    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:01.984344 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:01.984301    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:01.984595 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:01.984383    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:07:01.984595 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:01.984482    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:07:01.984595 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:01.984575    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:07:03.987862 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:03.987832    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:03.988372 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:03.987885    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:03.988372 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:03.987891    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:03.988372 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:03.987960    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:07:03.988372 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:03.988054    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:07:03.988372 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:03.988115    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:07:05.176387 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.175980    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5xj9x" event={"ID":"67f097ec-5a12-42f3-84bf-a66968a7a323","Type":"ContainerStarted","Data":"0a36028aa96ff3b41dc751f8f12f173035f62998b878421982556fdbc35e42bd"}
Sep 17 23:07:05.177660 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.177623    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rjcdn" event={"ID":"6d35f966-2cf2-46cc-9069-e971900f7430","Type":"ContainerStarted","Data":"f9ea486208539bb6ad6444976f40a08241292faf442b5321188ea915d7d455e0"}
Sep 17 23:07:05.179351 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.179323    2568 generic.go:356] "Generic (PLEG): container finished" podID="13483042-c3be-4ef8-b615-72852899eade" containerID="2607adb822a439a51308067e9f27ec3525de5194a994102588e87e49730c4ef7" exitCode=0
Sep 17 23:07:05.179454 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.179390    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sjsw" event={"ID":"13483042-c3be-4ef8-b615-72852899eade","Type":"ContainerDied","Data":"2607adb822a439a51308067e9f27ec3525de5194a994102588e87e49730c4ef7"}
Sep 17 23:07:05.180976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.180958    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xg9wq" event={"ID":"3560e9bb-709c-4dfc-9407-85406aec3f3e","Type":"ContainerStarted","Data":"e29b694627caed3fa1057a12a4dfb533977e96bf2832f76935013bdcaffba0e2"}
Sep 17 23:07:05.185972 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.185953    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj" event={"ID":"13416a44-5328-4fa3-9d5e-83a2ab0ad655","Type":"ContainerStarted","Data":"7b7e773c9114ab14ac53e9469364400672112fd24b0df625cd01a34e2be88a58"}
Sep 17 23:07:05.186070 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.186025    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj" event={"ID":"13416a44-5328-4fa3-9d5e-83a2ab0ad655","Type":"ContainerStarted","Data":"e89076333f968604cb666cdcd32bd1aed0b46b33106de039b7baa386fb65e610"}
Sep 17 23:07:05.186158 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.186077    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj" event={"ID":"13416a44-5328-4fa3-9d5e-83a2ab0ad655","Type":"ContainerStarted","Data":"29a5a43238d8ce90cd01a57306bce52e7f3cd10a5a2919075398d95f8ae46820"}
Sep 17 23:07:05.186158 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.186125    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj" event={"ID":"13416a44-5328-4fa3-9d5e-83a2ab0ad655","Type":"ContainerStarted","Data":"20a697f470909b3464b7074323246546a3ec90dd1c6339a513c3fd2cc8f95ef2"}
Sep 17 23:07:05.186236 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.186171    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj" event={"ID":"13416a44-5328-4fa3-9d5e-83a2ab0ad655","Type":"ContainerStarted","Data":"d4b309b5faa085c522395e470e050defc043584a04ec199d01df9a0b714a39d5"}
Sep 17 23:07:05.187360 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.187331    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qkccv" event={"ID":"2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa","Type":"ContainerStarted","Data":"556ca688dcbe02fce792dcb638228bba53051e58526323953a2f69916cb13d4e"}
Sep 17 23:07:05.188809 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.188789    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dghj8" event={"ID":"f537520e-e943-4837-a871-3142a15f6fba","Type":"ContainerStarted","Data":"4564324afa84ab7bf696fc18e928404ac5c11a4b4868e849be198fe1ec54e352"}
Sep 17 23:07:05.190359 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.190339    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6" event={"ID":"f70c2877-9d27-4ac0-8453-85018507bdef","Type":"ContainerStarted","Data":"f32bcbc5ad261b4b719492d7d594f96e3d483b9816357c512234fc5e25490d31"}
Sep 17 23:07:05.191181 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.191141    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5xj9x" podStartSLOduration=11.234311053 podStartE2EDuration="21.191128293s" totalImagesPullingTime="9.95681724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:05.190509031 +0000 UTC m=+23.414184779" watchObservedRunningTime="2026-09-17 23:07:05.191128293 +0000 UTC m=+23.414804034"
Sep 17 23:07:05.205479 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.205439    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dghj8" podStartSLOduration=3.762556961 podStartE2EDuration="21.205426025s" totalImagesPullingTime="17.442869064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:05.205070976 +0000 UTC m=+23.428746747" watchObservedRunningTime="2026-09-17 23:07:05.205426025 +0000 UTC m=+23.429101774"
Sep 17 23:07:05.222432 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.222389    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xg9wq" podStartSLOduration=3.74045768 podStartE2EDuration="21.222374084s" totalImagesPullingTime="17.481916404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:05.222242851 +0000 UTC m=+23.445918599" watchObservedRunningTime="2026-09-17 23:07:05.222374084 +0000 UTC m=+23.446049832"
Sep 17 23:07:05.237529 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.237484    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qkccv" podStartSLOduration=3.798088854 podStartE2EDuration="21.237468569s" totalImagesPullingTime="17.439379715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:05.236900608 +0000 UTC m=+23.460576358" watchObservedRunningTime="2026-09-17 23:07:05.237468569 +0000 UTC m=+23.461144317"
Sep 17 23:07:05.303691 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.303642    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rjcdn" podStartSLOduration=3.862992458 podStartE2EDuration="21.303626328s" totalImagesPullingTime="17.44063387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:05.303083666 +0000 UTC m=+23.526759415" watchObservedRunningTime="2026-09-17 23:07:05.303626328 +0000 UTC m=+23.527302075"
Sep 17 23:07:05.557441 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.557389    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:05.764855 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.764816    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:05.983811 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.983772    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:05.983976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.983772    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:05.983976 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:05.983922    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:05.983976 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:05.983941    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:07:05.984155 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:05.984031    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:07:05.984155 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:05.984098    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:07:06.009110 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:06.009085    2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 23:07:06.194897 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:06.194786    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xzrtv" event={"ID":"8be07c34-232d-4784-a1b4-0172240ebf7d","Type":"ContainerStarted","Data":"d27b0882b85798da5170f9242c218017d93e74b28a7a6d2ce666e56bc89a8192"}
Sep 17 23:07:06.198520 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:06.198451    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj" event={"ID":"13416a44-5328-4fa3-9d5e-83a2ab0ad655","Type":"ContainerStarted","Data":"70056d8f74281ba393e04f04241d4a8bbdf40882d4c0d44b5bd8db76bcb23b5d"}
Sep 17 23:07:06.200453 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:06.200425    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6" event={"ID":"f70c2877-9d27-4ac0-8453-85018507bdef","Type":"ContainerStarted","Data":"33036d92e70c251f54111cf5afbde06082561d545f02532d0b5303a6f0029546"}
Sep 17 23:07:06.210489 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:06.210437    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xzrtv" podStartSLOduration=4.7988113089999995 podStartE2EDuration="22.210420626s" totalImagesPullingTime="17.411609317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:06.210155618 +0000 UTC m=+24.433831365" watchObservedRunningTime="2026-09-17 23:07:06.210420626 +0000 UTC m=+24.434096374"
Sep 17 23:07:06.355424 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:06.355390    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:06.554173 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:06.554129    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:06.889548 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:06.889504    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5xj9x"
Sep 17 23:07:06.936066 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:06.935958    2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T23:07:06.009107596Z","UUID":"e68e1c02-3bee-4c7f-9c6d-474a4b987e55","Handler":null,"Name":"","Endpoint":""}
Sep 17 23:07:06.937633 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:06.937611    2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 23:07:06.937756 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:06.937649    2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 23:07:07.204803 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:07.204717    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6" event={"ID":"f70c2877-9d27-4ac0-8453-85018507bdef","Type":"ContainerStarted","Data":"68c520952f68cec2e27ee02fa1175e4dff61665df26c6f4e8ca939dea99235f4"}
Sep 17 23:07:07.222038 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:07.221979    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5gdc6" podStartSLOduration=3.97395967 podStartE2EDuration="23.221956811s" totalImagesPullingTime="19.247997141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:07.221275585 +0000 UTC m=+25.444951332" watchObservedRunningTime="2026-09-17 23:07:07.221956811 +0000 UTC m=+25.445632556"
Sep 17 23:07:07.226467 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:07.226443    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:07.356474 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:07.356438    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:07.415135 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:07.415104    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:07.644749 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:07.644716    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:07.753015 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:07.752979    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:07.857110 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:07.857077    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:07.916815 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:07.916730    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:07.984241 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:07.984209    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:07.984409 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:07.984336    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:07:07.984409 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:07.984209    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:07.984483 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:07.984422    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:07:07.984564 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:07.984544    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:07.984691 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:07.984613    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:07:08.152498 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:08.152454    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:08.210958 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:08.210824    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj" event={"ID":"13416a44-5328-4fa3-9d5e-83a2ab0ad655","Type":"ContainerStarted","Data":"8f850c5e851e756c7aa16b51df9b4a728a08c9dc6b4982cf1c32b3259a355c69"}
Sep 17 23:07:08.210958 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:08.210915    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:08.431086 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:08.431053    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:08.528530 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:08.528495    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5xj9x"
Sep 17 23:07:08.529081 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:08.529058    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5xj9x"
Sep 17 23:07:08.954380 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:08.954304    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:09.213696 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:09.213629    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5xj9x"
Sep 17 23:07:09.354700 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:09.354667    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:09.524164 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:09.523743    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:09.650788 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:09.650761    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:09.754096 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:09.754074    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:09.855284 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:09.855264    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:09.956263 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:09.956228    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:09.984319 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:09.984290    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:09.984476 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:09.984290    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:09.984476 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:09.984429    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:07:09.984476 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:09.984290    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:09.984614 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:09.984513    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:07:09.984614 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:09.984553    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:07:10.180510 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.180475    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:10.216936 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.216902    2568 generic.go:356] "Generic (PLEG): container finished" podID="13483042-c3be-4ef8-b615-72852899eade" containerID="0f2f29167420da7483f354f6b26f930fc8c3e546f614e1d0ae45dee9825e42d8" exitCode=0
Sep 17 23:07:10.217774 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.216990    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sjsw" event={"ID":"13483042-c3be-4ef8-b615-72852899eade","Type":"ContainerDied","Data":"0f2f29167420da7483f354f6b26f930fc8c3e546f614e1d0ae45dee9825e42d8"}
Sep 17 23:07:10.220841 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.220816    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj" event={"ID":"13416a44-5328-4fa3-9d5e-83a2ab0ad655","Type":"ContainerStarted","Data":"b41d66ed44827ab6d8f158d124c8233a95aead922da5888c687f4d0beba3a8df"}
Sep 17 23:07:10.221146 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.221123    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:07:10.221242 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.221191    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:07:10.236384 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.236359    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:07:10.236529 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.236471    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:07:10.266017 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.265969    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj" podStartSLOduration=8.828113678 podStartE2EDuration="26.265951795s" totalImagesPullingTime="17.437838117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:10.26545553 +0000 UTC m=+28.489131301" watchObservedRunningTime="2026-09-17 23:07:10.265951795 +0000 UTC m=+28.489627543"
Sep 17 23:07:10.294644 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.294602    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:10.454205 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.454126    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:10.519052 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.519016    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:10.687154 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.687124    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:10.855597 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.855562    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:10.916877 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.916833    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:10.973853 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:10.973820    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:11.222728 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:11.222661    2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 23:07:11.393042 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:11.393004    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:11.554794 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:11.554765    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:11.665386 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:11.665354    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:11.766677 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:11.766600    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:11.884949 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:11.884915    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:11.984608 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:11.984569    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:11.984780 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:11.984573    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:11.984780 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:11.984675    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:07:11.984780 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:11.984695    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:11.984780 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:11.984773    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:07:11.984976 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:11.984828    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:07:11.985722 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:11.985703    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:12.086434 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:12.086400    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:12.106179 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:12.106155    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:07:12.186071 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:12.186042    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:12.226915 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:12.226883    2568 generic.go:356] "Generic (PLEG): container finished" podID="13483042-c3be-4ef8-b615-72852899eade" containerID="503ae04d1890f57de037eeb40cd610608938e3b271d30b19b1701feec2c0952c" exitCode=0
Sep 17 23:07:12.227358 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:12.226960    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sjsw" event={"ID":"13483042-c3be-4ef8-b615-72852899eade","Type":"ContainerDied","Data":"503ae04d1890f57de037eeb40cd610608938e3b271d30b19b1701feec2c0952c"}
Sep 17 23:07:12.553680 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:12.553645    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:13.152552 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:13.152513    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:13.547510 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:13.547328    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:13.949330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:13.949291    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:13.984381 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:13.984344    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:13.984571 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:13.984447    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:07:13.984571 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:13.984469    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:13.984678 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:13.984569    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:07:13.984678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:13.984627    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:13.984758 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:13.984682    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:07:14.234137 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:14.234034    2568 generic.go:356] "Generic (PLEG): container finished" podID="13483042-c3be-4ef8-b615-72852899eade" containerID="c43f6a2000239336a95ad7d8e3e98670a13913b03011863e073fc4039d90cb9c" exitCode=0
Sep 17 23:07:14.234137 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:14.234071    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sjsw" event={"ID":"13483042-c3be-4ef8-b615-72852899eade","Type":"ContainerDied","Data":"c43f6a2000239336a95ad7d8e3e98670a13913b03011863e073fc4039d90cb9c"}
Sep 17 23:07:14.352880 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:14.352845    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:14.557593 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:14.557558    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:14.750064 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:14.750026    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:15.984157 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:15.984124    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:15.984595 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:15.984269    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:07:15.984595 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:15.984121    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:15.984595 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:15.984121    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:15.984595 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:15.984360    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:07:15.984595 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:15.984420    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:07:17.609399 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:17.609352    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:17.610025 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:17.609463    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs\") pod \"network-metrics-daemon-r5mz6\" (UID: \"76c81e49-6a97-4454-945f-9fd9bea45db2\") " pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:17.610025 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:17.609528    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:07:17.610025 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:17.609566    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:07:17.610025 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:17.609613    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret podName:75be2122-acf1-43e5-902d-c1d574d42792 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:49.609596628 +0000 UTC m=+67.833272354 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret") pod "global-pull-secret-syncer-rw5hp" (UID: "75be2122-acf1-43e5-902d-c1d574d42792") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:07:17.610025 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:17.609636    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs podName:76c81e49-6a97-4454-945f-9fd9bea45db2 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:49.609624181 +0000 UTC m=+67.833299907 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs") pod "network-metrics-daemon-r5mz6" (UID: "76c81e49-6a97-4454-945f-9fd9bea45db2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:07:17.810566 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:17.810526    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncd8\" (UniqueName: \"kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8\") pod \"network-check-target-9vxwv\" (UID: \"9e17f611-03ca-4632-938e-9af066f3bc1c\") " pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:17.810753 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:17.810692    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:07:17.810753 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:17.810736    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:07:17.810753 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:17.810749    2568 projected.go:196] Error preparing data for projected volume kube-api-access-vncd8 for pod openshift-network-diagnostics/network-check-target-9vxwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:07:17.810938 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:17.810817    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8 podName:9e17f611-03ca-4632-938e-9af066f3bc1c nodeName:}" failed. No retries permitted until 2026-09-17 23:07:49.810795577 +0000 UTC m=+68.034471323 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vncd8" (UniqueName: "kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8") pod "network-check-target-9vxwv" (UID: "9e17f611-03ca-4632-938e-9af066f3bc1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:07:17.984001 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:17.983917    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:17.984170 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:17.983926    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:17.984170 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:17.984047    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:07:17.984170 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:17.983922    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:17.984170 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:17.984139    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:07:17.984361 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:17.984201    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:07:19.248136 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:19.248105    2568 generic.go:356] "Generic (PLEG): container finished" podID="13483042-c3be-4ef8-b615-72852899eade" containerID="1f946ae6ec333a9e7b25fd159917c3887b613a67763ea8272cdbdc7308691b8f" exitCode=0
Sep 17 23:07:19.248514 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:19.248164    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sjsw" event={"ID":"13483042-c3be-4ef8-b615-72852899eade","Type":"ContainerDied","Data":"1f946ae6ec333a9e7b25fd159917c3887b613a67763ea8272cdbdc7308691b8f"}
Sep 17 23:07:19.489906 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:19.489861    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:19.984349 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:19.984311    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:19.984519 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:19.984312    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:19.984519 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:19.984421    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:07:19.984519 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:19.984312    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:19.984519 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:19.984508    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:07:19.984649 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:19.984588    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:07:20.253089 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:20.253001    2568 generic.go:356] "Generic (PLEG): container finished" podID="13483042-c3be-4ef8-b615-72852899eade" containerID="b04e37ebcc290f0bfeaa6008d9d0c94b7fbbe02699d794f5ef4de2f3bd16f96b" exitCode=0
Sep 17 23:07:20.253089 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:20.253053    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sjsw" event={"ID":"13483042-c3be-4ef8-b615-72852899eade","Type":"ContainerDied","Data":"b04e37ebcc290f0bfeaa6008d9d0c94b7fbbe02699d794f5ef4de2f3bd16f96b"}
Sep 17 23:07:21.248425 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:21.248390    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:21.258490 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:21.258462    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9sjsw" event={"ID":"13483042-c3be-4ef8-b615-72852899eade","Type":"ContainerStarted","Data":"df58583957d7e6f7cfc5d0957c25f2ffff3ffb5590bc8d99ffc764587d903f68"}
Sep 17 23:07:21.299486 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:21.299437    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9sjsw" podStartSLOduration=8.890519245 podStartE2EDuration="37.299420909s" totalImagesPullingTime="28.408901664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:21.298088299 +0000 UTC m=+39.521764046" watchObservedRunningTime="2026-09-17 23:07:21.299420909 +0000 UTC m=+39.523096657"
Sep 17 23:07:21.464166 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:21.464122    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:21.628318 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:21.628288    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:21.984081 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:21.983996    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:21.984245 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:21.983997    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:21.984245 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:21.984121    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:07:21.984245 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:21.984199    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:07:21.984245 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:21.983996    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:21.984456 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:21.984295    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:07:22.430255 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:22.430220    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:22.505569 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:22.505538    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:22.606040 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:22.606013    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:22.706627 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:22.706548    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:22.739001 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:22.737951    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r5mz6"]
Sep 17 23:07:22.739001 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:22.738166    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:22.739001 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:22.738330    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:07:22.742357 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:22.742322    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rw5hp"]
Sep 17 23:07:22.742507 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:22.742448    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:22.742616 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:22.742566    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:07:22.742998 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:22.742974    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9vxwv"]
Sep 17 23:07:22.743103 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:22.743062    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:22.743170 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:22.743147    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:07:22.789513 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:22.789480    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:22.877699 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:22.877663    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:22.976910 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:22.976667    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:23.984531 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:23.984452    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:23.984970 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:23.984542    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:07:23.984970 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:23.984577    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:23.984970 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:23.984620    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:07:24.983951 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:24.983921    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:24.984114 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:24.984029    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rw5hp" podUID="75be2122-acf1-43e5-902d-c1d574d42792"
Sep 17 23:07:25.984025 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:25.983988    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:25.984025 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:25.984024    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:25.984450 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:25.984113    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r5mz6" podUID="76c81e49-6a97-4454-945f-9fd9bea45db2"
Sep 17 23:07:25.984450 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:25.984225    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9vxwv" podUID="9e17f611-03ca-4632-938e-9af066f3bc1c"
Sep 17 23:07:26.647663 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.647630    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-44.ec2.internal" event="NodeReady"
Sep 17 23:07:26.647835 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.647789    2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 23:07:26.693384 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.693330    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4sv94"]
Sep 17 23:07:26.695170 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.695154    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wcn4x"]
Sep 17 23:07:26.695303 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.695286    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:26.697367 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.697348    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vvv69\""
Sep 17 23:07:26.697507 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.697350    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 23:07:26.697507 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.697446    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.697639 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.697616    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 23:07:26.699596 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.699569    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 23:07:26.699922 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.699905    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 23:07:26.700080 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.700062    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ktswr\""
Sep 17 23:07:26.700080 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.700075    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 23:07:26.700751 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.700735    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 23:07:26.707831 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.707815    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4sv94"]
Sep 17 23:07:26.708629 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.708610    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wcn4x"]
Sep 17 23:07:26.716792 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.716773    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wkjrp"]
Sep 17 23:07:26.718706 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.718692    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wkjrp"
Sep 17 23:07:26.721031 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.721015    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-672mp\""
Sep 17 23:07:26.721111 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.721097    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 23:07:26.721156 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.721114    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 23:07:26.721250 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.721234    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 23:07:26.729071 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.729053    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wkjrp"]
Sep 17 23:07:26.775838 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.775090    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be91cd5-c0cc-4166-9d53-fb219bf61469-metrics-tls\") pod \"dns-default-4sv94\" (UID: \"1be91cd5-c0cc-4166-9d53-fb219bf61469\") " pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:26.775838 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.775149    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1be91cd5-c0cc-4166-9d53-fb219bf61469-config-volume\") pod \"dns-default-4sv94\" (UID: \"1be91cd5-c0cc-4166-9d53-fb219bf61469\") " pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:26.775838 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.775265    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1be91cd5-c0cc-4166-9d53-fb219bf61469-tmp-dir\") pod \"dns-default-4sv94\" (UID: \"1be91cd5-c0cc-4166-9d53-fb219bf61469\") " pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:26.775838 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.775290    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5d9h\" (UniqueName: \"kubernetes.io/projected/1be91cd5-c0cc-4166-9d53-fb219bf61469-kube-api-access-n5d9h\") pod \"dns-default-4sv94\" (UID: \"1be91cd5-c0cc-4166-9d53-fb219bf61469\") " pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:26.841128 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.841096    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:26.876431 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.876363    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/677bf714-7a8a-4ce3-8610-5d304d16131e-cert\") pod \"ingress-canary-wkjrp\" (UID: \"677bf714-7a8a-4ce3-8610-5d304d16131e\") " pod="openshift-ingress-canary/ingress-canary-wkjrp"
Sep 17 23:07:26.876431 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.876403    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89ac468a-64c9-4576-a4a6-998e922cae73-data-volume\") pod \"insights-runtime-extractor-wcn4x\" (UID: \"89ac468a-64c9-4576-a4a6-998e922cae73\") " pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.876431 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.876430    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlks2\" (UniqueName: \"kubernetes.io/projected/677bf714-7a8a-4ce3-8610-5d304d16131e-kube-api-access-rlks2\") pod \"ingress-canary-wkjrp\" (UID: \"677bf714-7a8a-4ce3-8610-5d304d16131e\") " pod="openshift-ingress-canary/ingress-canary-wkjrp"
Sep 17 23:07:26.876586 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.876502    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1be91cd5-c0cc-4166-9d53-fb219bf61469-tmp-dir\") pod \"dns-default-4sv94\" (UID: \"1be91cd5-c0cc-4166-9d53-fb219bf61469\") " pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:26.876586 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.876521    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5d9h\" (UniqueName: \"kubernetes.io/projected/1be91cd5-c0cc-4166-9d53-fb219bf61469-kube-api-access-n5d9h\") pod \"dns-default-4sv94\" (UID: \"1be91cd5-c0cc-4166-9d53-fb219bf61469\") " pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:26.876586 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.876545    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89ac468a-64c9-4576-a4a6-998e922cae73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcn4x\" (UID: \"89ac468a-64c9-4576-a4a6-998e922cae73\") " pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.876586 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.876567    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89ac468a-64c9-4576-a4a6-998e922cae73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcn4x\" (UID: \"89ac468a-64c9-4576-a4a6-998e922cae73\") " pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.876743 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.876597    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89ac468a-64c9-4576-a4a6-998e922cae73-crio-socket\") pod \"insights-runtime-extractor-wcn4x\" (UID: \"89ac468a-64c9-4576-a4a6-998e922cae73\") " pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.876743 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.876628    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be91cd5-c0cc-4166-9d53-fb219bf61469-metrics-tls\") pod \"dns-default-4sv94\" (UID: \"1be91cd5-c0cc-4166-9d53-fb219bf61469\") " pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:26.876743 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.876648    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94blx\" (UniqueName: \"kubernetes.io/projected/89ac468a-64c9-4576-a4a6-998e922cae73-kube-api-access-94blx\") pod \"insights-runtime-extractor-wcn4x\" (UID: \"89ac468a-64c9-4576-a4a6-998e922cae73\") " pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.876743 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.876688    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1be91cd5-c0cc-4166-9d53-fb219bf61469-config-volume\") pod \"dns-default-4sv94\" (UID: \"1be91cd5-c0cc-4166-9d53-fb219bf61469\") " pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:26.876934 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.876914    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1be91cd5-c0cc-4166-9d53-fb219bf61469-tmp-dir\") pod \"dns-default-4sv94\" (UID: \"1be91cd5-c0cc-4166-9d53-fb219bf61469\") " pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:26.877174 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.877159    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1be91cd5-c0cc-4166-9d53-fb219bf61469-config-volume\") pod \"dns-default-4sv94\" (UID: \"1be91cd5-c0cc-4166-9d53-fb219bf61469\") " pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:26.880499 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.880480    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1be91cd5-c0cc-4166-9d53-fb219bf61469-metrics-tls\") pod \"dns-default-4sv94\" (UID: \"1be91cd5-c0cc-4166-9d53-fb219bf61469\") " pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:26.886685 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.886660    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5d9h\" (UniqueName: \"kubernetes.io/projected/1be91cd5-c0cc-4166-9d53-fb219bf61469-kube-api-access-n5d9h\") pod \"dns-default-4sv94\" (UID: \"1be91cd5-c0cc-4166-9d53-fb219bf61469\") " pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:26.935742 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.935706    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:26.977913 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.977859    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94blx\" (UniqueName: \"kubernetes.io/projected/89ac468a-64c9-4576-a4a6-998e922cae73-kube-api-access-94blx\") pod \"insights-runtime-extractor-wcn4x\" (UID: \"89ac468a-64c9-4576-a4a6-998e922cae73\") " pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.978032 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.977943    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/677bf714-7a8a-4ce3-8610-5d304d16131e-cert\") pod \"ingress-canary-wkjrp\" (UID: \"677bf714-7a8a-4ce3-8610-5d304d16131e\") " pod="openshift-ingress-canary/ingress-canary-wkjrp"
Sep 17 23:07:26.978032 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.977965    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89ac468a-64c9-4576-a4a6-998e922cae73-data-volume\") pod \"insights-runtime-extractor-wcn4x\" (UID: \"89ac468a-64c9-4576-a4a6-998e922cae73\") " pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.978032 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.977987    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlks2\" (UniqueName: \"kubernetes.io/projected/677bf714-7a8a-4ce3-8610-5d304d16131e-kube-api-access-rlks2\") pod \"ingress-canary-wkjrp\" (UID: \"677bf714-7a8a-4ce3-8610-5d304d16131e\") " pod="openshift-ingress-canary/ingress-canary-wkjrp"
Sep 17 23:07:26.978032 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.978024    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89ac468a-64c9-4576-a4a6-998e922cae73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcn4x\" (UID: \"89ac468a-64c9-4576-a4a6-998e922cae73\") " pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.978216 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.978047    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89ac468a-64c9-4576-a4a6-998e922cae73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcn4x\" (UID: \"89ac468a-64c9-4576-a4a6-998e922cae73\") " pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.978216 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.978075    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89ac468a-64c9-4576-a4a6-998e922cae73-crio-socket\") pod \"insights-runtime-extractor-wcn4x\" (UID: \"89ac468a-64c9-4576-a4a6-998e922cae73\") " pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.978301 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.978273    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89ac468a-64c9-4576-a4a6-998e922cae73-crio-socket\") pod \"insights-runtime-extractor-wcn4x\" (UID: \"89ac468a-64c9-4576-a4a6-998e922cae73\") " pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.978430 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.978411    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89ac468a-64c9-4576-a4a6-998e922cae73-data-volume\") pod \"insights-runtime-extractor-wcn4x\" (UID: \"89ac468a-64c9-4576-a4a6-998e922cae73\") " pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.978688 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.978664    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89ac468a-64c9-4576-a4a6-998e922cae73-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wcn4x\" (UID: \"89ac468a-64c9-4576-a4a6-998e922cae73\") " pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.980474 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.980447    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89ac468a-64c9-4576-a4a6-998e922cae73-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wcn4x\" (UID: \"89ac468a-64c9-4576-a4a6-998e922cae73\") " pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.980603 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.980587    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/677bf714-7a8a-4ce3-8610-5d304d16131e-cert\") pod \"ingress-canary-wkjrp\" (UID: \"677bf714-7a8a-4ce3-8610-5d304d16131e\") " pod="openshift-ingress-canary/ingress-canary-wkjrp"
Sep 17 23:07:26.983972 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.983949    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:26.986132 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.986107    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:07:26.986529 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.986444    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94blx\" (UniqueName: \"kubernetes.io/projected/89ac468a-64c9-4576-a4a6-998e922cae73-kube-api-access-94blx\") pod \"insights-runtime-extractor-wcn4x\" (UID: \"89ac468a-64c9-4576-a4a6-998e922cae73\") " pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:26.986529 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:26.986447    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlks2\" (UniqueName: \"kubernetes.io/projected/677bf714-7a8a-4ce3-8610-5d304d16131e-kube-api-access-rlks2\") pod \"ingress-canary-wkjrp\" (UID: \"677bf714-7a8a-4ce3-8610-5d304d16131e\") " pod="openshift-ingress-canary/ingress-canary-wkjrp"
Sep 17 23:07:27.006413 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:27.006394    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:27.012276 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:27.012261    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wcn4x"
Sep 17 23:07:27.027366 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:27.027344    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wkjrp"
Sep 17 23:07:27.294952 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:27.294915    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4sv94"]
Sep 17 23:07:27.297660 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:27.297630    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wcn4x"]
Sep 17 23:07:27.298980 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:07:27.298953    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be91cd5_c0cc_4166_9d53_fb219bf61469.slice/crio-bc0261a3ff4e84fa775fd85cf6c78dc4354bd8ee7a4ab77f9001fe65b05a8c5f WatchSource:0}: Error finding container bc0261a3ff4e84fa775fd85cf6c78dc4354bd8ee7a4ab77f9001fe65b05a8c5f: Status 404 returned error can't find the container with id bc0261a3ff4e84fa775fd85cf6c78dc4354bd8ee7a4ab77f9001fe65b05a8c5f
Sep 17 23:07:27.300537 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:07:27.300510    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ac468a_64c9_4576_a4a6_998e922cae73.slice/crio-160c32432701a646ecfd270858e23088e53c2942ca0da00172200b9ee4bc2807 WatchSource:0}: Error finding container 160c32432701a646ecfd270858e23088e53c2942ca0da00172200b9ee4bc2807: Status 404 returned error can't find the container with id 160c32432701a646ecfd270858e23088e53c2942ca0da00172200b9ee4bc2807
Sep 17 23:07:27.311127 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:27.310955    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wkjrp"]
Sep 17 23:07:27.314630 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:07:27.314610    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677bf714_7a8a_4ce3_8610_5d304d16131e.slice/crio-afd79cbf6f1da0d4df4a0bc54be622f4af3c852464598be9c20ae1efda41cc74 WatchSource:0}: Error finding container afd79cbf6f1da0d4df4a0bc54be622f4af3c852464598be9c20ae1efda41cc74: Status 404 returned error can't find the container with id afd79cbf6f1da0d4df4a0bc54be622f4af3c852464598be9c20ae1efda41cc74
Sep 17 23:07:27.984021 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:27.983987    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:27.984227 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:27.984036    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:27.987846 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:27.987816    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:07:27.988590 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:27.987981    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t46xp\""
Sep 17 23:07:27.988590 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:27.987822    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:07:27.988590 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:27.988208    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kxqn4\""
Sep 17 23:07:27.988590 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:27.988389    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:07:28.278539 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:28.278499    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wkjrp" event={"ID":"677bf714-7a8a-4ce3-8610-5d304d16131e","Type":"ContainerStarted","Data":"afd79cbf6f1da0d4df4a0bc54be622f4af3c852464598be9c20ae1efda41cc74"}
Sep 17 23:07:28.281169 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:28.281039    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wcn4x" event={"ID":"89ac468a-64c9-4576-a4a6-998e922cae73","Type":"ContainerStarted","Data":"d6332c477edf4c8653035f94b44cf8748c66b82d6dcff944e2bfcca7fc0c04e5"}
Sep 17 23:07:28.281169 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:28.281104    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wcn4x" event={"ID":"89ac468a-64c9-4576-a4a6-998e922cae73","Type":"ContainerStarted","Data":"8c27fd9ceef31472a37c4be821363fe728df4333879ba5f4bd214ea4e9d785fb"}
Sep 17 23:07:28.281169 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:28.281133    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wcn4x" event={"ID":"89ac468a-64c9-4576-a4a6-998e922cae73","Type":"ContainerStarted","Data":"160c32432701a646ecfd270858e23088e53c2942ca0da00172200b9ee4bc2807"}
Sep 17 23:07:28.282590 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:28.282556    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4sv94" event={"ID":"1be91cd5-c0cc-4166-9d53-fb219bf61469","Type":"ContainerStarted","Data":"bc0261a3ff4e84fa775fd85cf6c78dc4354bd8ee7a4ab77f9001fe65b05a8c5f"}
Sep 17 23:07:29.838057 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:29.837977    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:29.904463 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:29.904428    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:30.291711 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.291651    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:30.291923 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.291836    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wcn4x" event={"ID":"89ac468a-64c9-4576-a4a6-998e922cae73","Type":"ContainerStarted","Data":"dfeb9baeaf89cdbdb9d6d44a9052b5cb39a5bc4f36df58f0070680921d8bb931"}
Sep 17 23:07:30.293667 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.293640    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4sv94" event={"ID":"1be91cd5-c0cc-4166-9d53-fb219bf61469","Type":"ContainerStarted","Data":"be4542afa14c72b1901619dd6ca3c802302891836bababfef5781ad5d4a55840"}
Sep 17 23:07:30.293767 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.293687    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4sv94" event={"ID":"1be91cd5-c0cc-4166-9d53-fb219bf61469","Type":"ContainerStarted","Data":"13101dd6ccf8041203359b9a03f45ca92d915aeaf7a5bb925899ae1d8f042569"}
Sep 17 23:07:30.293767 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.293739    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:30.295062 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.295041    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wkjrp" event={"ID":"677bf714-7a8a-4ce3-8610-5d304d16131e","Type":"ContainerStarted","Data":"5f551d1133116bd6e1722a1c5aa9d0fd7fef52ee98c6da9c39ab47e0ddb634dd"}
Sep 17 23:07:30.316457 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.316411    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wcn4x" podStartSLOduration=1.817846287 podStartE2EDuration="4.316396827s" totalImagesPullingTime="2.49855054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:07:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:30.315125937 +0000 UTC m=+48.538801685" watchObservedRunningTime="2026-09-17 23:07:30.316396827 +0000 UTC m=+48.540072574"
Sep 17 23:07:30.331118 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.331071    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4sv94" podStartSLOduration=1.70712494 podStartE2EDuration="4.33104644s" totalImagesPullingTime="2.6239215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:30.33056003 +0000 UTC m=+48.554235778" watchObservedRunningTime="2026-09-17 23:07:30.33104644 +0000 UTC m=+48.554722188"
Sep 17 23:07:30.347402 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.347316    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wkjrp" podStartSLOduration=1.7389394729999998 podStartE2EDuration="4.34729483s" totalImagesPullingTime="2.608355357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:30.346831703 +0000 UTC m=+48.570507451" watchObservedRunningTime="2026-09-17 23:07:30.34729483 +0000 UTC m=+48.570970580"
Sep 17 23:07:30.888473 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.888395    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fd479bccb-8dszg"]
Sep 17 23:07:30.890610 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.890594    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:30.894250 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.894087    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 23:07:30.894250 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.894089    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 23:07:30.894250 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.894121    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 23:07:30.894250 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.894115    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 23:07:30.894250 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.894148    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 23:07:30.894250 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.894218    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 23:07:30.894629 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.894492    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 23:07:30.894731 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.894716    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rftn6\""
Sep 17 23:07:30.898926 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.898908    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 23:07:30.903208 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:30.903187    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd479bccb-8dszg"]
Sep 17 23:07:31.008727 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.008670    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-service-ca\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.008727 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.008729    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cf77948-c8eb-4f15-9021-5be425641cec-console-oauth-config\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.008977 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.008753    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-trusted-ca-bundle\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.008977 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.008816    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-console-config\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.008977 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.008845    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk9qt\" (UniqueName: \"kubernetes.io/projected/4cf77948-c8eb-4f15-9021-5be425641cec-kube-api-access-gk9qt\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.008977 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.008949    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-oauth-serving-cert\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.009102 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.008996    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cf77948-c8eb-4f15-9021-5be425641cec-console-serving-cert\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.110110 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.110056    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cf77948-c8eb-4f15-9021-5be425641cec-console-serving-cert\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.110110 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.110110    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-service-ca\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.110330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.110234    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cf77948-c8eb-4f15-9021-5be425641cec-console-oauth-config\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.110330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.110273    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-trusted-ca-bundle\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.110330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.110311    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-console-config\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.110330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.110328    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk9qt\" (UniqueName: \"kubernetes.io/projected/4cf77948-c8eb-4f15-9021-5be425641cec-kube-api-access-gk9qt\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.110457 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.110371    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-oauth-serving-cert\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.111512 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.111488    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-console-config\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.111589 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.111546    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-oauth-serving-cert\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.111637 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.111546    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-service-ca\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.111877 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.111851    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-trusted-ca-bundle\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.114675 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.114656    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cf77948-c8eb-4f15-9021-5be425641cec-console-serving-cert\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.114763 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.114659    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cf77948-c8eb-4f15-9021-5be425641cec-console-oauth-config\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.119843 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.119813    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk9qt\" (UniqueName: \"kubernetes.io/projected/4cf77948-c8eb-4f15-9021-5be425641cec-kube-api-access-gk9qt\") pod \"console-fd479bccb-8dszg\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") " pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.199773 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.199676    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:31.472115 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:31.472036    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd479bccb-8dszg"]
Sep 17 23:07:31.476040 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:07:31.475804    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf77948_c8eb_4f15_9021_5be425641cec.slice/crio-e2bbe2734738c221eb031834fd90a084275912686f52b912d1063a31a26ecaea WatchSource:0}: Error finding container e2bbe2734738c221eb031834fd90a084275912686f52b912d1063a31a26ecaea: Status 404 returned error can't find the container with id e2bbe2734738c221eb031834fd90a084275912686f52b912d1063a31a26ecaea
Sep 17 23:07:32.295716 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:32.295676    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:32.309178 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:32.309007    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd479bccb-8dszg" event={"ID":"4cf77948-c8eb-4f15-9021-5be425641cec","Type":"ContainerStarted","Data":"e2bbe2734738c221eb031834fd90a084275912686f52b912d1063a31a26ecaea"}
Sep 17 23:07:32.364682 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:32.364587    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:33.297950 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:33.297910    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:33.406045 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:33.406015    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:33.498481 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:33.498445    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:33.621504 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:33.621424    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:33.625441 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:33.625416    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-nlgxb"]
Sep 17 23:07:33.636675 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:33.636651    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-nlgxb"
Sep 17 23:07:33.639099 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:33.639075    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-nlgxb"]
Sep 17 23:07:33.639369 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:33.639349    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gks4r\""
Sep 17 23:07:33.639803 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:33.639654    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 23:07:33.807975 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:33.807930    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:33.833144 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:33.833107    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e5dd9ad2-27a9-4a50-9796-8d82601b1228-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-nlgxb\" (UID: \"e5dd9ad2-27a9-4a50-9796-8d82601b1228\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-nlgxb"
Sep 17 23:07:33.933520 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:33.933443    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e5dd9ad2-27a9-4a50-9796-8d82601b1228-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-nlgxb\" (UID: \"e5dd9ad2-27a9-4a50-9796-8d82601b1228\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-nlgxb"
Sep 17 23:07:33.937713 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:33.937678    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e5dd9ad2-27a9-4a50-9796-8d82601b1228-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-nlgxb\" (UID: \"e5dd9ad2-27a9-4a50-9796-8d82601b1228\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-nlgxb"
Sep 17 23:07:33.949055 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:33.949027    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-nlgxb"
Sep 17 23:07:34.296464 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:34.296429    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:34.356056 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:34.356022    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:34.655823 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:34.655736    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:34.759693 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:34.759640    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:34.820545 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:34.820512    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:35.021031 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:35.020948    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:35.125690 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:35.125646    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:35.232660 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:35.232623    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:35.501567 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:35.501522    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:35.556659 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:35.556629    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:35.994108 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:35.994077    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-nlgxb"]
Sep 17 23:07:36.039471 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:07:36.039427    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5dd9ad2_27a9_4a50_9796_8d82601b1228.slice/crio-3d44810d497e9c730529f90f7f86f0f6de997346a4faf684e5b065f134db92d8 WatchSource:0}: Error finding container 3d44810d497e9c730529f90f7f86f0f6de997346a4faf684e5b065f134db92d8: Status 404 returned error can't find the container with id 3d44810d497e9c730529f90f7f86f0f6de997346a4faf684e5b065f134db92d8
Sep 17 23:07:36.326275 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:36.326003    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd479bccb-8dszg" event={"ID":"4cf77948-c8eb-4f15-9021-5be425641cec","Type":"ContainerStarted","Data":"5bf5b8ac33afe6930f52833fc3d2ad334bfcdd02d49ac0d9dec0cc56a0aa2bd2"}
Sep 17 23:07:36.327264 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:36.327233    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-nlgxb" event={"ID":"e5dd9ad2-27a9-4a50-9796-8d82601b1228","Type":"ContainerStarted","Data":"3d44810d497e9c730529f90f7f86f0f6de997346a4faf684e5b065f134db92d8"}
Sep 17 23:07:36.344897 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:36.344804    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fd479bccb-8dszg" podStartSLOduration=1.749171509 podStartE2EDuration="6.344789144s" totalImagesPullingTime="4.595617635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:36.34452165 +0000 UTC m=+54.568197399" watchObservedRunningTime="2026-09-17 23:07:36.344789144 +0000 UTC m=+54.568464891"
Sep 17 23:07:37.888362 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:37.888329    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:37.985409 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:37.985375    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:38.335371 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.335332    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-nlgxb" event={"ID":"e5dd9ad2-27a9-4a50-9796-8d82601b1228","Type":"ContainerStarted","Data":"6ba765e3e0dfb4d923ce954e468d1ab3755b47e59011e0244d7bd3786359b801"}
Sep 17 23:07:38.335573 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.335553    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-nlgxb"
Sep 17 23:07:38.340143 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.340121    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-nlgxb"
Sep 17 23:07:38.354539 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.354492    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-nlgxb" podStartSLOduration=4.023978629 podStartE2EDuration="5.354475586s" totalImagesPullingTime="1.330496957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:38.353921223 +0000 UTC m=+56.577596972" watchObservedRunningTime="2026-09-17 23:07:38.354475586 +0000 UTC m=+56.578151334"
Sep 17 23:07:38.513226 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.513193    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:38.652261 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.652231    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:38.693367 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.693330    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"]
Sep 17 23:07:38.706800 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.706771    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"
Sep 17 23:07:38.708295 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.708272    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"]
Sep 17 23:07:38.709831 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.709804    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 23:07:38.711583 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.710538    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 23:07:38.711583 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.710723    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 23:07:38.711583 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.711072    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 23:07:38.711583 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.711292    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gfpcw\""
Sep 17 23:07:38.711583 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.711384    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 23:07:38.733247 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.733220    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:38.767548 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.767512    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3415eddf-d7d2-4171-8269-5ca12702a674-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-n2rtl\" (UID: \"3415eddf-d7d2-4171-8269-5ca12702a674\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"
Sep 17 23:07:38.767665 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.767574    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3415eddf-d7d2-4171-8269-5ca12702a674-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-n2rtl\" (UID: \"3415eddf-d7d2-4171-8269-5ca12702a674\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"
Sep 17 23:07:38.767665 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.767596    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg66f\" (UniqueName: \"kubernetes.io/projected/3415eddf-d7d2-4171-8269-5ca12702a674-kube-api-access-dg66f\") pod \"prometheus-operator-7bd5d56d6-n2rtl\" (UID: \"3415eddf-d7d2-4171-8269-5ca12702a674\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"
Sep 17 23:07:38.767665 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.767631    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3415eddf-d7d2-4171-8269-5ca12702a674-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-n2rtl\" (UID: \"3415eddf-d7d2-4171-8269-5ca12702a674\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"
Sep 17 23:07:38.868023 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.867934    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3415eddf-d7d2-4171-8269-5ca12702a674-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-n2rtl\" (UID: \"3415eddf-d7d2-4171-8269-5ca12702a674\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"
Sep 17 23:07:38.868023 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.867971    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg66f\" (UniqueName: \"kubernetes.io/projected/3415eddf-d7d2-4171-8269-5ca12702a674-kube-api-access-dg66f\") pod \"prometheus-operator-7bd5d56d6-n2rtl\" (UID: \"3415eddf-d7d2-4171-8269-5ca12702a674\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"
Sep 17 23:07:38.868023 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.868009    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3415eddf-d7d2-4171-8269-5ca12702a674-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-n2rtl\" (UID: \"3415eddf-d7d2-4171-8269-5ca12702a674\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"
Sep 17 23:07:38.868264 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.868051    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3415eddf-d7d2-4171-8269-5ca12702a674-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-n2rtl\" (UID: \"3415eddf-d7d2-4171-8269-5ca12702a674\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"
Sep 17 23:07:38.868264 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:38.868158    2568 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 23:07:38.868264 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:38.868223    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3415eddf-d7d2-4171-8269-5ca12702a674-prometheus-operator-tls podName:3415eddf-d7d2-4171-8269-5ca12702a674 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:39.368207729 +0000 UTC m=+57.591883455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/3415eddf-d7d2-4171-8269-5ca12702a674-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-n2rtl" (UID: "3415eddf-d7d2-4171-8269-5ca12702a674") : secret "prometheus-operator-tls" not found
Sep 17 23:07:38.868759 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.868741    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3415eddf-d7d2-4171-8269-5ca12702a674-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-n2rtl\" (UID: \"3415eddf-d7d2-4171-8269-5ca12702a674\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"
Sep 17 23:07:38.871731 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.871711    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3415eddf-d7d2-4171-8269-5ca12702a674-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-n2rtl\" (UID: \"3415eddf-d7d2-4171-8269-5ca12702a674\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"
Sep 17 23:07:38.877082 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.877056    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg66f\" (UniqueName: \"kubernetes.io/projected/3415eddf-d7d2-4171-8269-5ca12702a674-kube-api-access-dg66f\") pod \"prometheus-operator-7bd5d56d6-n2rtl\" (UID: \"3415eddf-d7d2-4171-8269-5ca12702a674\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"
Sep 17 23:07:38.955593 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:38.955560    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:39.019070 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:39.019035    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:39.370810 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:39.370775    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3415eddf-d7d2-4171-8269-5ca12702a674-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-n2rtl\" (UID: \"3415eddf-d7d2-4171-8269-5ca12702a674\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"
Sep 17 23:07:39.373125 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:39.373099    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3415eddf-d7d2-4171-8269-5ca12702a674-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-n2rtl\" (UID: \"3415eddf-d7d2-4171-8269-5ca12702a674\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"
Sep 17 23:07:39.618560 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:39.618522    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"
Sep 17 23:07:39.915046 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:39.914964    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl"]
Sep 17 23:07:39.917825 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:07:39.917794    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3415eddf_d7d2_4171_8269_5ca12702a674.slice/crio-b70d7a603f3631f7676377bbb55618e1c991c895f173bda9237052133cdd5619 WatchSource:0}: Error finding container b70d7a603f3631f7676377bbb55618e1c991c895f173bda9237052133cdd5619: Status 404 returned error can't find the container with id b70d7a603f3631f7676377bbb55618e1c991c895f173bda9237052133cdd5619
Sep 17 23:07:40.304009 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:40.303980    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4sv94"
Sep 17 23:07:40.342285 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:40.342247    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl" event={"ID":"3415eddf-d7d2-4171-8269-5ca12702a674","Type":"ContainerStarted","Data":"b70d7a603f3631f7676377bbb55618e1c991c895f173bda9237052133cdd5619"}
Sep 17 23:07:40.934883 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:40.934835    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:41.027265 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:41.027233    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:41.200769 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:41.200676    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:41.200769 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:41.200749    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:41.205627 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:41.205602    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:41.349249 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:41.349221    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:07:41.515083 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:41.515010    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:41.615215 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:41.615183    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:41.715452 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:41.715413    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:41.834602 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:41.834565    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:41.942036 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:41.942000    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:42.008343 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:42.008314    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:42.349668 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:42.349639    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl" event={"ID":"3415eddf-d7d2-4171-8269-5ca12702a674","Type":"ContainerStarted","Data":"770503c720c17fab18563795333a0758364be69418da417bcae7b5bf7a708003"}
Sep 17 23:07:42.350084 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:42.349686    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl" event={"ID":"3415eddf-d7d2-4171-8269-5ca12702a674","Type":"ContainerStarted","Data":"d49a3b55ae1b329d02228094df780a81007b98feb3841d791524751fb71d4c1d"}
Sep 17 23:07:42.367184 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:42.367132    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-n2rtl" podStartSLOduration=2.763192777 podStartE2EDuration="4.367113927s" totalImagesPullingTime="1.60392115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:42.366527858 +0000 UTC m=+60.590203606" watchObservedRunningTime="2026-09-17 23:07:42.367113927 +0000 UTC m=+60.590789675"
Sep 17 23:07:43.240563 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:43.240536    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zlpxj"
Sep 17 23:07:43.393321 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:43.393289    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:43.453123 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:43.453093    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:43.795437 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:43.795403    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:43.889754 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:43.889718    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:44.012771 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.012738    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:44.036272 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.036242    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"]
Sep 17 23:07:44.041250 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.041221    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"
Sep 17 23:07:44.043751 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.043721    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 23:07:44.043905 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.043837    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 23:07:44.043977 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.043902    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sh2rg\""
Sep 17 23:07:44.050149 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.050129    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"]
Sep 17 23:07:44.056025 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.056001    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"]
Sep 17 23:07:44.058174 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.058155    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.060895 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.060830    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 23:07:44.061102 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.061085    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rqxrh\""
Sep 17 23:07:44.061102 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.061095    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 23:07:44.061238 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.061122    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 23:07:44.075454 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.075425    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"]
Sep 17 23:07:44.085941 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.085916    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-55dvl"]
Sep 17 23:07:44.088076 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.088062    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.090382 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.090362    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 23:07:44.090702 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.090677    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5h2qn\""
Sep 17 23:07:44.090812 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.090679    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 23:07:44.090812 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.090712    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 23:07:44.101694 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.101668    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9blt\" (UniqueName: \"kubernetes.io/projected/80ef4d7a-3b32-4de6-b515-61762093f346-kube-api-access-r9blt\") pod \"openshift-state-metrics-86c5bc79d6-jwstr\" (UID: \"80ef4d7a-3b32-4de6-b515-61762093f346\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"
Sep 17 23:07:44.101822 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.101705    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a866547-9807-43ca-86bd-a50a9d7d0f30-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.101822 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.101732    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd5f437d-1f12-49f2-bfeb-0dda727c740b-sys\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.101822 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.101770    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd5f437d-1f12-49f2-bfeb-0dda727c740b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.101956 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.101838    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26mvv\" (UniqueName: \"kubernetes.io/projected/1a866547-9807-43ca-86bd-a50a9d7d0f30-kube-api-access-26mvv\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.101956 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.101862    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80ef4d7a-3b32-4de6-b515-61762093f346-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-jwstr\" (UID: \"80ef4d7a-3b32-4de6-b515-61762093f346\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"
Sep 17 23:07:44.101956 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.101904    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1a866547-9807-43ca-86bd-a50a9d7d0f30-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.101956 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.101919    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd5f437d-1f12-49f2-bfeb-0dda727c740b-node-exporter-tls\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.101956 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.101948    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80ef4d7a-3b32-4de6-b515-61762093f346-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-jwstr\" (UID: \"80ef4d7a-3b32-4de6-b515-61762093f346\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"
Sep 17 23:07:44.102117 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.102016    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cd5f437d-1f12-49f2-bfeb-0dda727c740b-node-exporter-wtmp\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.102117 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.102039    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80ef4d7a-3b32-4de6-b515-61762093f346-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-jwstr\" (UID: \"80ef4d7a-3b32-4de6-b515-61762093f346\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"
Sep 17 23:07:44.102117 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.102061    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a866547-9807-43ca-86bd-a50a9d7d0f30-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.102117 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.102090    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a866547-9807-43ca-86bd-a50a9d7d0f30-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.102233 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.102119    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1a866547-9807-43ca-86bd-a50a9d7d0f30-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.102233 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.102149    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cd5f437d-1f12-49f2-bfeb-0dda727c740b-node-exporter-textfile\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.102233 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.102167    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps6jz\" (UniqueName: \"kubernetes.io/projected/cd5f437d-1f12-49f2-bfeb-0dda727c740b-kube-api-access-ps6jz\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.102233 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.102203    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cd5f437d-1f12-49f2-bfeb-0dda727c740b-root\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.102233 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.102216    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd5f437d-1f12-49f2-bfeb-0dda727c740b-metrics-client-ca\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.102380 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.102252    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cd5f437d-1f12-49f2-bfeb-0dda727c740b-node-exporter-accelerators-collector-config\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.108498 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.108480    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:44.203037 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203000    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1a866547-9807-43ca-86bd-a50a9d7d0f30-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.203037 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203037    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd5f437d-1f12-49f2-bfeb-0dda727c740b-node-exporter-tls\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.203287 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203068    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80ef4d7a-3b32-4de6-b515-61762093f346-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-jwstr\" (UID: \"80ef4d7a-3b32-4de6-b515-61762093f346\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"
Sep 17 23:07:44.203287 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203100    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cd5f437d-1f12-49f2-bfeb-0dda727c740b-node-exporter-wtmp\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.203287 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203124    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80ef4d7a-3b32-4de6-b515-61762093f346-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-jwstr\" (UID: \"80ef4d7a-3b32-4de6-b515-61762093f346\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"
Sep 17 23:07:44.203287 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203151    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a866547-9807-43ca-86bd-a50a9d7d0f30-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.203287 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203195    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a866547-9807-43ca-86bd-a50a9d7d0f30-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.203287 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203228    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1a866547-9807-43ca-86bd-a50a9d7d0f30-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.203287 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203253    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cd5f437d-1f12-49f2-bfeb-0dda727c740b-node-exporter-textfile\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.203287 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203281    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps6jz\" (UniqueName: \"kubernetes.io/projected/cd5f437d-1f12-49f2-bfeb-0dda727c740b-kube-api-access-ps6jz\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.203689 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:44.203299    2568 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 17 23:07:44.203689 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203313    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cd5f437d-1f12-49f2-bfeb-0dda727c740b-root\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.203689 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203339    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd5f437d-1f12-49f2-bfeb-0dda727c740b-metrics-client-ca\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.203689 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203364    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cd5f437d-1f12-49f2-bfeb-0dda727c740b-node-exporter-accelerators-collector-config\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.203689 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203411    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9blt\" (UniqueName: \"kubernetes.io/projected/80ef4d7a-3b32-4de6-b515-61762093f346-kube-api-access-r9blt\") pod \"openshift-state-metrics-86c5bc79d6-jwstr\" (UID: \"80ef4d7a-3b32-4de6-b515-61762093f346\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"
Sep 17 23:07:44.203689 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203440    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a866547-9807-43ca-86bd-a50a9d7d0f30-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.203689 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:07:44.203462    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ef4d7a-3b32-4de6-b515-61762093f346-openshift-state-metrics-tls podName:80ef4d7a-3b32-4de6-b515-61762093f346 nodeName:}" failed. No retries permitted until 2026-09-17 23:07:44.703439063 +0000 UTC m=+62.927114799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/80ef4d7a-3b32-4de6-b515-61762093f346-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-jwstr" (UID: "80ef4d7a-3b32-4de6-b515-61762093f346") : secret "openshift-state-metrics-tls" not found
Sep 17 23:07:44.203689 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203522    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd5f437d-1f12-49f2-bfeb-0dda727c740b-sys\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.203689 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203558    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd5f437d-1f12-49f2-bfeb-0dda727c740b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.203689 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203635    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26mvv\" (UniqueName: \"kubernetes.io/projected/1a866547-9807-43ca-86bd-a50a9d7d0f30-kube-api-access-26mvv\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.203689 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203639    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1a866547-9807-43ca-86bd-a50a9d7d0f30-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.203689 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203673    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80ef4d7a-3b32-4de6-b515-61762093f346-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-jwstr\" (UID: \"80ef4d7a-3b32-4de6-b515-61762093f346\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"
Sep 17 23:07:44.204354 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203838    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1a866547-9807-43ca-86bd-a50a9d7d0f30-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.204354 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203932    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd5f437d-1f12-49f2-bfeb-0dda727c740b-sys\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.204354 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203942    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80ef4d7a-3b32-4de6-b515-61762093f346-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-jwstr\" (UID: \"80ef4d7a-3b32-4de6-b515-61762093f346\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"
Sep 17 23:07:44.204354 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203948    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cd5f437d-1f12-49f2-bfeb-0dda727c740b-root\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.204354 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.203297    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cd5f437d-1f12-49f2-bfeb-0dda727c740b-node-exporter-wtmp\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.204758 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.204733    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cd5f437d-1f12-49f2-bfeb-0dda727c740b-node-exporter-textfile\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.205008 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.204983    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd5f437d-1f12-49f2-bfeb-0dda727c740b-metrics-client-ca\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.205075 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.205028    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a866547-9807-43ca-86bd-a50a9d7d0f30-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.205122 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.205065    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cd5f437d-1f12-49f2-bfeb-0dda727c740b-node-exporter-accelerators-collector-config\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.206149 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.206125    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a866547-9807-43ca-86bd-a50a9d7d0f30-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.206245 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.206149    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd5f437d-1f12-49f2-bfeb-0dda727c740b-node-exporter-tls\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.206317 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.206287    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/80ef4d7a-3b32-4de6-b515-61762093f346-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-jwstr\" (UID: \"80ef4d7a-3b32-4de6-b515-61762093f346\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"
Sep 17 23:07:44.206317 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.206287    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a866547-9807-43ca-86bd-a50a9d7d0f30-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.207030 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.207009    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd5f437d-1f12-49f2-bfeb-0dda727c740b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.216027 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.215998    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26mvv\" (UniqueName: \"kubernetes.io/projected/1a866547-9807-43ca-86bd-a50a9d7d0f30-kube-api-access-26mvv\") pod \"kube-state-metrics-84c885658c-tg8gr\" (UID: \"1a866547-9807-43ca-86bd-a50a9d7d0f30\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.216174 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.216154    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps6jz\" (UniqueName: \"kubernetes.io/projected/cd5f437d-1f12-49f2-bfeb-0dda727c740b-kube-api-access-ps6jz\") pod \"node-exporter-55dvl\" (UID: \"cd5f437d-1f12-49f2-bfeb-0dda727c740b\") " pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.216547 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.216527    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9blt\" (UniqueName: \"kubernetes.io/projected/80ef4d7a-3b32-4de6-b515-61762093f346-kube-api-access-r9blt\") pod \"openshift-state-metrics-86c5bc79d6-jwstr\" (UID: \"80ef4d7a-3b32-4de6-b515-61762093f346\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"
Sep 17 23:07:44.222379 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.222344    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:44.321544 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.321459    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:44.366456 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.366419    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"
Sep 17 23:07:44.396408 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.396361    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-55dvl"
Sep 17 23:07:44.406450 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:07:44.406420    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5f437d_1f12_49f2_bfeb_0dda727c740b.slice/crio-b4dfa4026a9e570e63858c6ef5362966261deaf9297d90e3971f862217acb8fa WatchSource:0}: Error finding container b4dfa4026a9e570e63858c6ef5362966261deaf9297d90e3971f862217acb8fa: Status 404 returned error can't find the container with id b4dfa4026a9e570e63858c6ef5362966261deaf9297d90e3971f862217acb8fa
Sep 17 23:07:44.422879 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.422820    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:44.579978 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.579893    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:44.638675 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.638643    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-tg8gr"]
Sep 17 23:07:44.641287 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:07:44.641265    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a866547_9807_43ca_86bd_a50a9d7d0f30.slice/crio-9b0ae42f20e170f0d65234043037920cfa0f8ce93713c99c9ae021289c208457 WatchSource:0}: Error finding container 9b0ae42f20e170f0d65234043037920cfa0f8ce93713c99c9ae021289c208457: Status 404 returned error can't find the container with id 9b0ae42f20e170f0d65234043037920cfa0f8ce93713c99c9ae021289c208457
Sep 17 23:07:44.707646 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.707592    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80ef4d7a-3b32-4de6-b515-61762093f346-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-jwstr\" (UID: \"80ef4d7a-3b32-4de6-b515-61762093f346\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"
Sep 17 23:07:44.709977 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.709957    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/80ef4d7a-3b32-4de6-b515-61762093f346-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-jwstr\" (UID: \"80ef4d7a-3b32-4de6-b515-61762093f346\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"
Sep 17 23:07:44.890040 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.889949    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:44.950052 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.950012    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"
Sep 17 23:07:44.996769 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:44.996649    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:45.097526 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:45.097488    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:45.171929 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:45.171841    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:45.227454 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:45.227275    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr"]
Sep 17 23:07:45.229555 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:45.229509    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:45.231797 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:07:45.231772    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ef4d7a_3b32_4de6_b515_61762093f346.slice/crio-9aa1cb4351257082c0d7f8375f178b79b6881a0a172f3f5945e098452ab98dcf WatchSource:0}: Error finding container 9aa1cb4351257082c0d7f8375f178b79b6881a0a172f3f5945e098452ab98dcf: Status 404 returned error can't find the container with id 9aa1cb4351257082c0d7f8375f178b79b6881a0a172f3f5945e098452ab98dcf
Sep 17 23:07:45.364148 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:45.364071    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-55dvl" event={"ID":"cd5f437d-1f12-49f2-bfeb-0dda727c740b","Type":"ContainerStarted","Data":"2b78959cd4266c98ad8cefab14f267d52af7f07d16b2670019c914c0f2a2f5fc"}
Sep 17 23:07:45.364148 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:45.364139    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-55dvl" event={"ID":"cd5f437d-1f12-49f2-bfeb-0dda727c740b","Type":"ContainerStarted","Data":"b4dfa4026a9e570e63858c6ef5362966261deaf9297d90e3971f862217acb8fa"}
Sep 17 23:07:45.365917 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:45.365836    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr" event={"ID":"1a866547-9807-43ca-86bd-a50a9d7d0f30","Type":"ContainerStarted","Data":"9b0ae42f20e170f0d65234043037920cfa0f8ce93713c99c9ae021289c208457"}
Sep 17 23:07:45.368561 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:45.368397    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr" event={"ID":"80ef4d7a-3b32-4de6-b515-61762093f346","Type":"ContainerStarted","Data":"41d652d0a280bcbd1ad4dc7b024f0f2a229cddeb46b3c5933c2bca1979145989"}
Sep 17 23:07:45.368561 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:45.368448    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr" event={"ID":"80ef4d7a-3b32-4de6-b515-61762093f346","Type":"ContainerStarted","Data":"9aa1cb4351257082c0d7f8375f178b79b6881a0a172f3f5945e098452ab98dcf"}
Sep 17 23:07:45.391086 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:45.391029    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:45.445788 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:45.445687    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:46.030456 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.030431    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-787444c6d5-86h6h"]
Sep 17 23:07:46.032875 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.032845    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.035475 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.035427    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 23:07:46.035475 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.035439    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 23:07:46.035708 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.035509    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-auorv46b00q7o\""
Sep 17 23:07:46.035708 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.035438    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 23:07:46.035708 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.035617    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8qpcj\""
Sep 17 23:07:46.035850 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.035780    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 23:07:46.035850 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.035841    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 23:07:46.046885 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.046823    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-787444c6d5-86h6h"]
Sep 17 23:07:46.121565 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.121533    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.121726 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.121591    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.121726 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.121630    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-thanos-querier-tls\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.121726 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.121652    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-grpc-tls\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.121726 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.121675    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpr2k\" (UniqueName: \"kubernetes.io/projected/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-kube-api-access-kpr2k\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.121965 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.121732    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.121965 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.121766    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.121965 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.121838    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-metrics-client-ca\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.223176 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.223130    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.223366 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.223192    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-thanos-querier-tls\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.223366 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.223223    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-grpc-tls\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.223366 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.223252    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpr2k\" (UniqueName: \"kubernetes.io/projected/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-kube-api-access-kpr2k\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.223366 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.223284    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.223366 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.223311    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.223366 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.223365    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-metrics-client-ca\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.223674 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.223449    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.224812 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.224782    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-metrics-client-ca\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.226399 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.226334    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.226938 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.226882    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.227306 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.227283    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.227802 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.227751    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-grpc-tls\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.228134 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.228107    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.229763 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.229741    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-secret-thanos-querier-tls\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.233123 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.233099    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpr2k\" (UniqueName: \"kubernetes.io/projected/fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9-kube-api-access-kpr2k\") pod \"thanos-querier-787444c6d5-86h6h\" (UID: \"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9\") " pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.342813 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.342783    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:46.374370 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.374334    2568 generic.go:356] "Generic (PLEG): container finished" podID="cd5f437d-1f12-49f2-bfeb-0dda727c740b" containerID="2b78959cd4266c98ad8cefab14f267d52af7f07d16b2670019c914c0f2a2f5fc" exitCode=0
Sep 17 23:07:46.374538 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.374415    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-55dvl" event={"ID":"cd5f437d-1f12-49f2-bfeb-0dda727c740b","Type":"ContainerDied","Data":"2b78959cd4266c98ad8cefab14f267d52af7f07d16b2670019c914c0f2a2f5fc"}
Sep 17 23:07:46.377128 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.377095    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr" event={"ID":"1a866547-9807-43ca-86bd-a50a9d7d0f30","Type":"ContainerStarted","Data":"67247f837665e396c6ae53c1e4de7e751e39e50bfc409b6856590ef5dc0ee9cf"}
Sep 17 23:07:46.377260 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.377161    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr" event={"ID":"1a866547-9807-43ca-86bd-a50a9d7d0f30","Type":"ContainerStarted","Data":"5c857f648a77d4408f932174edc4bf80349cdf4e93afbefaf7de5b374d0dcb34"}
Sep 17 23:07:46.377260 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.377184    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr" event={"ID":"1a866547-9807-43ca-86bd-a50a9d7d0f30","Type":"ContainerStarted","Data":"fd88bafc20bee531a765d3b522be42ce53a9cc9077a5ea5a00fa4398e16e2e42"}
Sep 17 23:07:46.379343 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.379316    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr" event={"ID":"80ef4d7a-3b32-4de6-b515-61762093f346","Type":"ContainerStarted","Data":"7971985a0ae389712594e49848900e51031e1587a61ee2587ef87a8ec8c4c029"}
Sep 17 23:07:46.433770 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.433712    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-tg8gr" podStartSLOduration=1.098387754 podStartE2EDuration="2.433689789s" totalImagesPullingTime="1.335302035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:46.432105659 +0000 UTC m=+64.655781408" watchObservedRunningTime="2026-09-17 23:07:46.433689789 +0000 UTC m=+64.657365538"
Sep 17 23:07:46.814755 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.814664    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-787444c6d5-86h6h"]
Sep 17 23:07:46.818440 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:07:46.818417    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2140a9_cfbc_43c2_b6ed_ab77fc6184d9.slice/crio-3789e1b57cab5786b71be9fe4e11975fb34f376dc761d5dcea51684fa02f636c WatchSource:0}: Error finding container 3789e1b57cab5786b71be9fe4e11975fb34f376dc761d5dcea51684fa02f636c: Status 404 returned error can't find the container with id 3789e1b57cab5786b71be9fe4e11975fb34f376dc761d5dcea51684fa02f636c
Sep 17 23:07:46.976352 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:46.976312    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:47.031791 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:47.031764    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:47.383893 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:47.383836    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h" event={"ID":"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9","Type":"ContainerStarted","Data":"3789e1b57cab5786b71be9fe4e11975fb34f376dc761d5dcea51684fa02f636c"}
Sep 17 23:07:47.385948 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:47.385918    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-55dvl" event={"ID":"cd5f437d-1f12-49f2-bfeb-0dda727c740b","Type":"ContainerStarted","Data":"490a52837a5a628f500bfc5b9f6db58273489a0320c18cba9177369477dcc2c8"}
Sep 17 23:07:47.386091 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:47.385975    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-55dvl" event={"ID":"cd5f437d-1f12-49f2-bfeb-0dda727c740b","Type":"ContainerStarted","Data":"b2c8619926a1ad1e6bfdb2de78566a7aba081662246ee4607a6eebde19ad9bc9"}
Sep 17 23:07:47.387949 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:47.387926    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr" event={"ID":"80ef4d7a-3b32-4de6-b515-61762093f346","Type":"ContainerStarted","Data":"bae0e5c8afb9d0211287c2b77602d97ea0480941cd127bc583c6b920ef1b426b"}
Sep 17 23:07:47.407708 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:47.407660    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-55dvl" podStartSLOduration=2.50361073 podStartE2EDuration="3.407646236s" totalImagesPullingTime="820.863688ms" totalInitContainerRuntime="83.171818ms" isStatefulPod=true podCreationTimestamp="2026-09-17 23:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:47.405910923 +0000 UTC m=+65.629586672" watchObservedRunningTime="2026-09-17 23:07:47.407646236 +0000 UTC m=+65.631321983"
Sep 17 23:07:47.421942 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:47.421889    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-jwstr" podStartSLOduration=2.208938554 podStartE2EDuration="3.421849601s" totalImagesPullingTime="1.212911047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:47.421724663 +0000 UTC m=+65.645400411" watchObservedRunningTime="2026-09-17 23:07:47.421849601 +0000 UTC m=+65.645525350"
Sep 17 23:07:47.485723 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:47.485655    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:47.584179 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:47.584137    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:48.814585 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:48.814554    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:48.849838 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:48.849804    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-695bd4fb54-cktj7"]
Sep 17 23:07:48.870224 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:48.870195    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-695bd4fb54-cktj7"]
Sep 17 23:07:48.870369 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:48.870311    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:48.891738 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:48.891710    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:48.949344 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:48.949311    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-service-ca\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:48.949494 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:48.949356    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffpcx\" (UniqueName: \"kubernetes.io/projected/d8771e5f-12f7-40b4-895d-3fa206648938-kube-api-access-ffpcx\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:48.949494 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:48.949377    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-oauth-serving-cert\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:48.949494 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:48.949445    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8771e5f-12f7-40b4-895d-3fa206648938-console-serving-cert\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:48.949604 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:48.949558    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-console-config\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:48.949604 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:48.949584    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8771e5f-12f7-40b4-895d-3fa206648938-console-oauth-config\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:48.949703 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:48.949623    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-trusted-ca-bundle\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:48.952931 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:48.952860    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:49.044115 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.044089    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:49.050308 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.050280    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-console-config\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:49.050391 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.050329    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8771e5f-12f7-40b4-895d-3fa206648938-console-oauth-config\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:49.050525 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.050502    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-trusted-ca-bundle\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:49.050578 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.050544    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-service-ca\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:49.050635 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.050619    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffpcx\" (UniqueName: \"kubernetes.io/projected/d8771e5f-12f7-40b4-895d-3fa206648938-kube-api-access-ffpcx\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:49.050696 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.050651    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-oauth-serving-cert\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:49.050696 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.050692    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8771e5f-12f7-40b4-895d-3fa206648938-console-serving-cert\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:49.051243 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.051221    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-service-ca\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:49.051243 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.051238    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-console-config\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:49.051405 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.051356    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-oauth-serving-cert\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:49.051828 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.051807    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-trusted-ca-bundle\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:49.053314 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.053294    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8771e5f-12f7-40b4-895d-3fa206648938-console-serving-cert\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:49.053668 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.053652    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8771e5f-12f7-40b4-895d-3fa206648938-console-oauth-config\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:49.059255 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.059230    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffpcx\" (UniqueName: \"kubernetes.io/projected/d8771e5f-12f7-40b4-895d-3fa206648938-kube-api-access-ffpcx\") pod \"console-695bd4fb54-cktj7\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") " pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:49.181179 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.181145    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:49.398845 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.398815    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h" event={"ID":"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9","Type":"ContainerStarted","Data":"1c7b945d70e4d54133fd734a9e73f5b3c04998ea3621f28f08b7dd98fedf3a0d"}
Sep 17 23:07:49.398981 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.398901    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h" event={"ID":"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9","Type":"ContainerStarted","Data":"13750f9c24b69eb8a5372d57fdfe541869b0d9bfeda847d7965d2a8791ee9526"}
Sep 17 23:07:49.484583 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.484398    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-695bd4fb54-cktj7"]
Sep 17 23:07:49.487073 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:07:49.487046    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8771e5f_12f7_40b4_895d_3fa206648938.slice/crio-2c39f21e03849c311afd72ea8928c5125aea6903a50f0926e10678216e9d60ef WatchSource:0}: Error finding container 2c39f21e03849c311afd72ea8928c5125aea6903a50f0926e10678216e9d60ef: Status 404 returned error can't find the container with id 2c39f21e03849c311afd72ea8928c5125aea6903a50f0926e10678216e9d60ef
Sep 17 23:07:49.656269 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.656180    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs\") pod \"network-metrics-daemon-r5mz6\" (UID: \"76c81e49-6a97-4454-945f-9fd9bea45db2\") " pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:49.656432 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.656343    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:49.658507 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.658485    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:07:49.658610 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.658554    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:07:49.668677 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.668655    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75be2122-acf1-43e5-902d-c1d574d42792-original-pull-secret\") pod \"global-pull-secret-syncer-rw5hp\" (UID: \"75be2122-acf1-43e5-902d-c1d574d42792\") " pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:49.668790 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.668703    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c81e49-6a97-4454-945f-9fd9bea45db2-metrics-certs\") pod \"network-metrics-daemon-r5mz6\" (UID: \"76c81e49-6a97-4454-945f-9fd9bea45db2\") " pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:49.798855 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.798819    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rw5hp"
Sep 17 23:07:49.858294 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.858252    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncd8\" (UniqueName: \"kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8\") pod \"network-check-target-9vxwv\" (UID: \"9e17f611-03ca-4632-938e-9af066f3bc1c\") " pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:49.862071 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.861133    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:07:49.871568 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.871534    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:07:49.882291 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.882147    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vncd8\" (UniqueName: \"kubernetes.io/projected/9e17f611-03ca-4632-938e-9af066f3bc1c-kube-api-access-vncd8\") pod \"network-check-target-9vxwv\" (UID: \"9e17f611-03ca-4632-938e-9af066f3bc1c\") " pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:49.901139 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.901112    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kxqn4\""
Sep 17 23:07:49.908155 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.908096    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t46xp\""
Sep 17 23:07:49.909100 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.909074    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:49.917031 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:49.916992    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r5mz6"
Sep 17 23:07:50.086126 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.085763    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rw5hp"]
Sep 17 23:07:50.104772 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:07:50.104731    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75be2122_acf1_43e5_902d_c1d574d42792.slice/crio-a81e6d654d1164ff15d6f563871d9e5fad4919e275123d746014e7dd8aeac400 WatchSource:0}: Error finding container a81e6d654d1164ff15d6f563871d9e5fad4919e275123d746014e7dd8aeac400: Status 404 returned error can't find the container with id a81e6d654d1164ff15d6f563871d9e5fad4919e275123d746014e7dd8aeac400
Sep 17 23:07:50.209088 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.209062    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9vxwv"]
Sep 17 23:07:50.212164 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:07:50.212141    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e17f611_03ca_4632_938e_9af066f3bc1c.slice/crio-40331ca1789f3f826ae670a3dec81db63dd08ecfb32a6c5b06d1435ff65c575b WatchSource:0}: Error finding container 40331ca1789f3f826ae670a3dec81db63dd08ecfb32a6c5b06d1435ff65c575b: Status 404 returned error can't find the container with id 40331ca1789f3f826ae670a3dec81db63dd08ecfb32a6c5b06d1435ff65c575b
Sep 17 23:07:50.284407 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.282437    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:07:50.284707 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.284684    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:50.286745 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.286724    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.290662 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.290592    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 23:07:50.292750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.290977    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 23:07:50.292750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.291037    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 23:07:50.292750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.291242    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 23:07:50.292750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.291315    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-54cvs\""
Sep 17 23:07:50.292750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.291417    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 23:07:50.292750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.291636    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8ftd8u70k8ep2\""
Sep 17 23:07:50.292750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.291651    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 23:07:50.292750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.291696    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 23:07:50.292750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.291903    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 23:07:50.292750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.291942    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 23:07:50.292750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.292136    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 23:07:50.292750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.292324    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 23:07:50.292750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.292657    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 23:07:50.293476 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.293087    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 23:07:50.293476 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.293232    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 23:07:50.294940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.294906    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 23:07:50.302429 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.302403    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:07:50.362928 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.362861    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/39b3f0ef-e598-4742-af00-180e9f1cd9d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363123 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.362942    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363123 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.362965    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363123 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.362991    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363123 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363015    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363123 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363065    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363123 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363104    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-config\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363123 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363124    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39b3f0ef-e598-4742-af00-180e9f1cd9d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363399 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363143    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363399 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363157    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363399 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363183    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363399 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363201    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phhn6\" (UniqueName: \"kubernetes.io/projected/39b3f0ef-e598-4742-af00-180e9f1cd9d2-kube-api-access-phhn6\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363399 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363281    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363399 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363323    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363399 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363353    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363399 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363387    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363771 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363413    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39b3f0ef-e598-4742-af00-180e9f1cd9d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363771 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363457    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363771 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363488    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.363771 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.363513    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.377648 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.377505    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r5mz6"]
Sep 17 23:07:50.380430 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:07:50.380402    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c81e49_6a97_4454_945f_9fd9bea45db2.slice/crio-78b727fd4a62250cdee84712201afa221753f129ff7327c5f183ec4259be0f65 WatchSource:0}: Error finding container 78b727fd4a62250cdee84712201afa221753f129ff7327c5f183ec4259be0f65: Status 404 returned error can't find the container with id 78b727fd4a62250cdee84712201afa221753f129ff7327c5f183ec4259be0f65
Sep 17 23:07:50.403805 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.403774    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9vxwv" event={"ID":"9e17f611-03ca-4632-938e-9af066f3bc1c","Type":"ContainerStarted","Data":"40331ca1789f3f826ae670a3dec81db63dd08ecfb32a6c5b06d1435ff65c575b"}
Sep 17 23:07:50.405300 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.405271    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rw5hp" event={"ID":"75be2122-acf1-43e5-902d-c1d574d42792","Type":"ContainerStarted","Data":"a81e6d654d1164ff15d6f563871d9e5fad4919e275123d746014e7dd8aeac400"}
Sep 17 23:07:50.409506 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.409457    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h" event={"ID":"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9","Type":"ContainerStarted","Data":"c597b3202559ef75bfe40db3d3a6c1b219381712a3d7eb9d7eeaa8578fe1cfcb"}
Sep 17 23:07:50.409641 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.409548    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h" event={"ID":"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9","Type":"ContainerStarted","Data":"9c07a9261f75b375774c591bf5e3c265c8dbe131e96a9de44bbca671ca50eaf9"}
Sep 17 23:07:50.409641 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.409605    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h" event={"ID":"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9","Type":"ContainerStarted","Data":"5210fec55a0e4fda145fd7196a08403403c89a19b759b09367f24605a4be8ea9"}
Sep 17 23:07:50.409747 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.409643    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h" event={"ID":"fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9","Type":"ContainerStarted","Data":"dae2a0456b5d8d14845873d9c3e6fe87e5873767187bf72c12d6128c57ab83ff"}
Sep 17 23:07:50.409862 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.409844    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:50.412003 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.411976    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695bd4fb54-cktj7" event={"ID":"d8771e5f-12f7-40b4-895d-3fa206648938","Type":"ContainerStarted","Data":"353874fb6d7a99f39c3abbaf075ae6c4e2873b0f24c2852e843d91b2a9e55293"}
Sep 17 23:07:50.412115 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.412022    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695bd4fb54-cktj7" event={"ID":"d8771e5f-12f7-40b4-895d-3fa206648938","Type":"ContainerStarted","Data":"2c39f21e03849c311afd72ea8928c5125aea6903a50f0926e10678216e9d60ef"}
Sep 17 23:07:50.413480 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.413456    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5mz6" event={"ID":"76c81e49-6a97-4454-945f-9fd9bea45db2","Type":"ContainerStarted","Data":"78b727fd4a62250cdee84712201afa221753f129ff7327c5f183ec4259be0f65"}
Sep 17 23:07:50.431851 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.431769    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:50.432231 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.432189    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h" podStartSLOduration=1.520200439 podStartE2EDuration="4.432174589s" totalImagesPullingTime="2.91197415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:50.429986856 +0000 UTC m=+68.653662605" watchObservedRunningTime="2026-09-17 23:07:50.432174589 +0000 UTC m=+68.655850338"
Sep 17 23:07:50.449053 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.448991    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-695bd4fb54-cktj7" podStartSLOduration=2.448974556 podStartE2EDuration="2.448974556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:50.448024637 +0000 UTC m=+68.671700387" watchObservedRunningTime="2026-09-17 23:07:50.448974556 +0000 UTC m=+68.672650306"
Sep 17 23:07:50.464278 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.464239    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.464278 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.464289    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39b3f0ef-e598-4742-af00-180e9f1cd9d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.464597 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.464369    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.464597 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.464399    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.464597 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.464426    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.464597 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.464454    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/39b3f0ef-e598-4742-af00-180e9f1cd9d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.465729 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.465700    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/39b3f0ef-e598-4742-af00-180e9f1cd9d2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.465729 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.465705    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.466014 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.465766    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.466412 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.466388    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.467896 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.467523    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.469122 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.468495    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.469122 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.468759    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.469122 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.469009    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.469122 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.469105    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.469122 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.469111    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39b3f0ef-e598-4742-af00-180e9f1cd9d2-config-out\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.469531 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.469306    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.469531 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.469347    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.469531 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.469438    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-config\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.469531 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.469485    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39b3f0ef-e598-4742-af00-180e9f1cd9d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.469531 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.469527    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.474108 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.469597    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.474108 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.469693    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.474108 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.469737    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.474108 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.469755    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phhn6\" (UniqueName: \"kubernetes.io/projected/39b3f0ef-e598-4742-af00-180e9f1cd9d2-kube-api-access-phhn6\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.474108 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.469823    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.474108 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.469899    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.474108 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.469931    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.474108 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.472248    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.474108 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.473153    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.474108 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.473461    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.474607 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.474322    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/39b3f0ef-e598-4742-af00-180e9f1cd9d2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.476023 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.474982    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.476023 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.475467    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.476023 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.475518    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.476572 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.476386    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-config\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.477369 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.477347    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.477508 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.477484    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39b3f0ef-e598-4742-af00-180e9f1cd9d2-web-config\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.477691 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.477674    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39b3f0ef-e598-4742-af00-180e9f1cd9d2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.484772 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.484741    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phhn6\" (UniqueName: \"kubernetes.io/projected/39b3f0ef-e598-4742-af00-180e9f1cd9d2-kube-api-access-phhn6\") pod \"prometheus-k8s-0\" (UID: \"39b3f0ef-e598-4742-af00-180e9f1cd9d2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.534641 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.534597    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:50.614740 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.614701    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:07:50.915672 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.915635    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:07:50.919218 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:07:50.919184    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b3f0ef_e598_4742_af00_180e9f1cd9d2.slice/crio-09aa6234f1d427c44e4b1655d6526cd5e256edd533630a5dd6fa4c1951c6f09f WatchSource:0}: Error finding container 09aa6234f1d427c44e4b1655d6526cd5e256edd533630a5dd6fa4c1951c6f09f: Status 404 returned error can't find the container with id 09aa6234f1d427c44e4b1655d6526cd5e256edd533630a5dd6fa4c1951c6f09f
Sep 17 23:07:50.982800 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:50.982759    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:51.084808 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:51.084690    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:51.421068 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:51.421029    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39b3f0ef-e598-4742-af00-180e9f1cd9d2","Type":"ContainerStarted","Data":"09aa6234f1d427c44e4b1655d6526cd5e256edd533630a5dd6fa4c1951c6f09f"}
Sep 17 23:07:52.428122 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:52.428076    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5mz6" event={"ID":"76c81e49-6a97-4454-945f-9fd9bea45db2","Type":"ContainerStarted","Data":"4c8800ffdb70ed103f89f3f02f8921fb9703149acc6d571ab41222b5d82eab41"}
Sep 17 23:07:52.428595 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:52.428163    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r5mz6" event={"ID":"76c81e49-6a97-4454-945f-9fd9bea45db2","Type":"ContainerStarted","Data":"9a9126506743d32f6a53240b80c2ee827e8d57f0c9eda9e3e1175fff48edb341"}
Sep 17 23:07:52.430145 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:52.430114    2568 generic.go:356] "Generic (PLEG): container finished" podID="39b3f0ef-e598-4742-af00-180e9f1cd9d2" containerID="1ec2fd1a21911e0598af455056fa0735c9b798b0743a1ea4fce3d299cd6b5b14" exitCode=0
Sep 17 23:07:52.430286 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:52.430170    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39b3f0ef-e598-4742-af00-180e9f1cd9d2","Type":"ContainerDied","Data":"1ec2fd1a21911e0598af455056fa0735c9b798b0743a1ea4fce3d299cd6b5b14"}
Sep 17 23:07:52.447284 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:52.446902    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r5mz6" podStartSLOduration=66.827678979 podStartE2EDuration="1m8.446878684s" totalImagesPullingTime="1.619199705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:52.44475065 +0000 UTC m=+70.668426398" watchObservedRunningTime="2026-09-17 23:07:52.446878684 +0000 UTC m=+70.670554428"
Sep 17 23:07:52.537047 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:52.537015    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:52.629721 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:52.629681    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:52.854428 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:52.854334    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:52.955259 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:52.955223    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:53.119402 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:53.119318    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:53.175737 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:53.175700    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:55.445788 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:55.445590    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9vxwv" event={"ID":"9e17f611-03ca-4632-938e-9af066f3bc1c","Type":"ContainerStarted","Data":"d908462272eead632e777f1ee17d3a05830955c3f93bc125868f50f671136756"}
Sep 17 23:07:55.446318 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:55.446223    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:07:55.448085 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:55.448055    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rw5hp" event={"ID":"75be2122-acf1-43e5-902d-c1d574d42792","Type":"ContainerStarted","Data":"b034875f7d6e630e276937fa7e77c96ab5c5fb5950e72a727dcbe3b882767595"}
Sep 17 23:07:55.462610 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:55.462552    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9vxwv" podStartSLOduration=66.649101718 podStartE2EDuration="1m11.46253246s" totalImagesPullingTime="4.813430742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:55.461147561 +0000 UTC m=+73.684823310" watchObservedRunningTime="2026-09-17 23:07:55.46253246 +0000 UTC m=+73.686208208"
Sep 17 23:07:56.429610 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:56.429568    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-787444c6d5-86h6h"
Sep 17 23:07:56.456454 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:56.456395    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rw5hp" podStartSLOduration=67.53193184 podStartE2EDuration="1m12.456373992s" totalImagesPullingTime="4.924442152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:55.476757117 +0000 UTC m=+73.700432869" watchObservedRunningTime="2026-09-17 23:07:56.456373992 +0000 UTC m=+74.680049740"
Sep 17 23:07:58.466185 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:58.466154    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39b3f0ef-e598-4742-af00-180e9f1cd9d2","Type":"ContainerStarted","Data":"355c0adcb6ab849675b2fa929ac53d8e1cca0580fb45142056294ccd3492d474"}
Sep 17 23:07:58.466556 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:58.466217    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39b3f0ef-e598-4742-af00-180e9f1cd9d2","Type":"ContainerStarted","Data":"1ea3b04d477bcc8b2be312613204c88fcf2e15abdec0281040ca5bf02946df42"}
Sep 17 23:07:58.466556 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:58.466243    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39b3f0ef-e598-4742-af00-180e9f1cd9d2","Type":"ContainerStarted","Data":"f741060ba66822705499c021ec5be87324ab66fe90587cda1ab594dbe9661cd1"}
Sep 17 23:07:58.466556 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:58.466266    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39b3f0ef-e598-4742-af00-180e9f1cd9d2","Type":"ContainerStarted","Data":"e3f1b01f935bcd237ab6f639d44fee9c761ae136f6172c81a2b6965f2b5194e5"}
Sep 17 23:07:58.466556 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:58.466291    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39b3f0ef-e598-4742-af00-180e9f1cd9d2","Type":"ContainerStarted","Data":"b4bd65207900f1168570b92afa5e9463325d7059c222f45c293435637e118be4"}
Sep 17 23:07:58.466556 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:58.466315    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39b3f0ef-e598-4742-af00-180e9f1cd9d2","Type":"ContainerStarted","Data":"f65a4181a5e7dfaac0852d15bd6de476523a08ca924cc1eeccccb39f7aa0ec8f"}
Sep 17 23:07:58.496302 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:58.496243    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.045223969 podStartE2EDuration="8.496219589s" totalImagesPullingTime="6.423714551s" totalInitContainerRuntime="27.281069ms" isStatefulPod=false podCreationTimestamp="2026-09-17 23:07:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:07:58.494940895 +0000 UTC m=+76.718616646" watchObservedRunningTime="2026-09-17 23:07:58.496219589 +0000 UTC m=+76.719895340"
Sep 17 23:07:59.181964 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:59.181907    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:59.181964 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:59.181979    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:59.187053 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:59.187030    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:59.472822 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:59.472744    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:59.474104 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:59.474069    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:07:59.521715 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:59.521680    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fd479bccb-8dszg"]
Sep 17 23:07:59.549741 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:59.549707    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:59.695426 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:59.695396    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:59.796733 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:59.796698    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:07:59.846629 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:07:59.846598    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:08:00.615130 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:00.615097    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:08:07.856220 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:07.856124    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:08:07.909349 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:07.909312    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:08:17.422304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:17.422262    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:08:17.475751 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:17.475711    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:08:24.541511 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.541444    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fd479bccb-8dszg" podUID="4cf77948-c8eb-4f15-9021-5be425641cec" containerName="console" containerID="cri-o://5bf5b8ac33afe6930f52833fc3d2ad334bfcdd02d49ac0d9dec0cc56a0aa2bd2" gracePeriod=15
Sep 17 23:08:24.772441 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.772416    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:08:24.871948 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.871828    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-service-ca\") pod \"4cf77948-c8eb-4f15-9021-5be425641cec\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") "
Sep 17 23:08:24.871948 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.871916    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cf77948-c8eb-4f15-9021-5be425641cec-console-serving-cert\") pod \"4cf77948-c8eb-4f15-9021-5be425641cec\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") "
Sep 17 23:08:24.871948 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.871951    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-trusted-ca-bundle\") pod \"4cf77948-c8eb-4f15-9021-5be425641cec\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") "
Sep 17 23:08:24.872223 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.871973    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cf77948-c8eb-4f15-9021-5be425641cec-console-oauth-config\") pod \"4cf77948-c8eb-4f15-9021-5be425641cec\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") "
Sep 17 23:08:24.872223 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.872012    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-console-config\") pod \"4cf77948-c8eb-4f15-9021-5be425641cec\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") "
Sep 17 23:08:24.872223 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.872043    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-oauth-serving-cert\") pod \"4cf77948-c8eb-4f15-9021-5be425641cec\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") "
Sep 17 23:08:24.872425 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.872386    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-service-ca" (OuterVolumeSpecName: "service-ca") pod "4cf77948-c8eb-4f15-9021-5be425641cec" (UID: "4cf77948-c8eb-4f15-9021-5be425641cec"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:08:24.872489 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.872419    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4cf77948-c8eb-4f15-9021-5be425641cec" (UID: "4cf77948-c8eb-4f15-9021-5be425641cec"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:08:24.872489 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.872448    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-console-config" (OuterVolumeSpecName: "console-config") pod "4cf77948-c8eb-4f15-9021-5be425641cec" (UID: "4cf77948-c8eb-4f15-9021-5be425641cec"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:08:24.872578 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.872487    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4cf77948-c8eb-4f15-9021-5be425641cec" (UID: "4cf77948-c8eb-4f15-9021-5be425641cec"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:08:24.874450 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.874416    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf77948-c8eb-4f15-9021-5be425641cec-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4cf77948-c8eb-4f15-9021-5be425641cec" (UID: "4cf77948-c8eb-4f15-9021-5be425641cec"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:08:24.874560 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.874501    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf77948-c8eb-4f15-9021-5be425641cec-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4cf77948-c8eb-4f15-9021-5be425641cec" (UID: "4cf77948-c8eb-4f15-9021-5be425641cec"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:08:24.972881 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.972844    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk9qt\" (UniqueName: \"kubernetes.io/projected/4cf77948-c8eb-4f15-9021-5be425641cec-kube-api-access-gk9qt\") pod \"4cf77948-c8eb-4f15-9021-5be425641cec\" (UID: \"4cf77948-c8eb-4f15-9021-5be425641cec\") "
Sep 17 23:08:24.973068 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.973022    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:08:24.973068 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.973034    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cf77948-c8eb-4f15-9021-5be425641cec-console-serving-cert\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:08:24.973068 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.973043    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-trusted-ca-bundle\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:08:24.973068 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.973051    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cf77948-c8eb-4f15-9021-5be425641cec-console-oauth-config\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:08:24.973068 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.973059    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-console-config\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:08:24.973068 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.973068    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cf77948-c8eb-4f15-9021-5be425641cec-oauth-serving-cert\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:08:24.974989 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:24.974963    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf77948-c8eb-4f15-9021-5be425641cec-kube-api-access-gk9qt" (OuterVolumeSpecName: "kube-api-access-gk9qt") pod "4cf77948-c8eb-4f15-9021-5be425641cec" (UID: "4cf77948-c8eb-4f15-9021-5be425641cec"). InnerVolumeSpecName "kube-api-access-gk9qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:08:25.074208 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:25.074152    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk9qt\" (UniqueName: \"kubernetes.io/projected/4cf77948-c8eb-4f15-9021-5be425641cec-kube-api-access-gk9qt\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:08:25.578275 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:25.578238    2568 generic.go:356] "Generic (PLEG): container finished" podID="4cf77948-c8eb-4f15-9021-5be425641cec" containerID="5bf5b8ac33afe6930f52833fc3d2ad334bfcdd02d49ac0d9dec0cc56a0aa2bd2" exitCode=0
Sep 17 23:08:25.578728 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:25.578308    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd479bccb-8dszg"
Sep 17 23:08:25.578728 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:25.578318    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd479bccb-8dszg" event={"ID":"4cf77948-c8eb-4f15-9021-5be425641cec","Type":"ContainerDied","Data":"5bf5b8ac33afe6930f52833fc3d2ad334bfcdd02d49ac0d9dec0cc56a0aa2bd2"}
Sep 17 23:08:25.578728 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:25.578365    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd479bccb-8dszg" event={"ID":"4cf77948-c8eb-4f15-9021-5be425641cec","Type":"ContainerDied","Data":"e2bbe2734738c221eb031834fd90a084275912686f52b912d1063a31a26ecaea"}
Sep 17 23:08:25.578728 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:25.578394    2568 scope.go:117] "RemoveContainer" containerID="5bf5b8ac33afe6930f52833fc3d2ad334bfcdd02d49ac0d9dec0cc56a0aa2bd2"
Sep 17 23:08:25.588975 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:25.588957    2568 scope.go:117] "RemoveContainer" containerID="5bf5b8ac33afe6930f52833fc3d2ad334bfcdd02d49ac0d9dec0cc56a0aa2bd2"
Sep 17 23:08:25.589290 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:08:25.589269    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf5b8ac33afe6930f52833fc3d2ad334bfcdd02d49ac0d9dec0cc56a0aa2bd2\": container with ID starting with 5bf5b8ac33afe6930f52833fc3d2ad334bfcdd02d49ac0d9dec0cc56a0aa2bd2 not found: ID does not exist" containerID="5bf5b8ac33afe6930f52833fc3d2ad334bfcdd02d49ac0d9dec0cc56a0aa2bd2"
Sep 17 23:08:25.589337 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:25.589300    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf5b8ac33afe6930f52833fc3d2ad334bfcdd02d49ac0d9dec0cc56a0aa2bd2"} err="failed to get container status \"5bf5b8ac33afe6930f52833fc3d2ad334bfcdd02d49ac0d9dec0cc56a0aa2bd2\": rpc error: code = NotFound desc = could not find container \"5bf5b8ac33afe6930f52833fc3d2ad334bfcdd02d49ac0d9dec0cc56a0aa2bd2\": container with ID starting with 5bf5b8ac33afe6930f52833fc3d2ad334bfcdd02d49ac0d9dec0cc56a0aa2bd2 not found: ID does not exist"
Sep 17 23:08:25.600526 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:25.600496    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fd479bccb-8dszg"]
Sep 17 23:08:25.604225 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:25.604203    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fd479bccb-8dszg"]
Sep 17 23:08:25.988939 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:25.988844    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cf77948-c8eb-4f15-9021-5be425641cec" path="/var/lib/kubelet/pods/4cf77948-c8eb-4f15-9021-5be425641cec/volumes"
Sep 17 23:08:27.462057 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:27.462022    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9vxwv"
Sep 17 23:08:50.615410 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:50.615368    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:08:50.634878 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:50.634838    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:08:50.688104 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:08:50.688077    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:09:03.251923 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:03.251855    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:03.452748 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:03.452717    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:11.357072 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:11.356991    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:11.408518 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:11.408482    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:11.498828 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:11.498796    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:11.878082 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:11.878044    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:12.108653 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:12.108619    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:18.509974 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:18.509935    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:18.611494 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:18.611459    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:22.273529 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:22.273495    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-695bd4fb54-cktj7"]
Sep 17 23:09:30.504699 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:30.504664    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:30.606154 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:30.606121    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:33.416027 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:33.415997    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:33.471708 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:33.471677    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:33.527075 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:33.527039    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:37.863319 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:37.863273    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:37.964621 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:37.964587    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:09:47.292833 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.292766    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-695bd4fb54-cktj7" podUID="d8771e5f-12f7-40b4-895d-3fa206648938" containerName="console" containerID="cri-o://353874fb6d7a99f39c3abbaf075ae6c4e2873b0f24c2852e843d91b2a9e55293" gracePeriod=15
Sep 17 23:09:47.534377 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.534353    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:09:47.629379 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.629288    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-trusted-ca-bundle\") pod \"d8771e5f-12f7-40b4-895d-3fa206648938\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") "
Sep 17 23:09:47.629379 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.629333    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-console-config\") pod \"d8771e5f-12f7-40b4-895d-3fa206648938\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") "
Sep 17 23:09:47.629379 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.629363    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-service-ca\") pod \"d8771e5f-12f7-40b4-895d-3fa206648938\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") "
Sep 17 23:09:47.629658 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.629394    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8771e5f-12f7-40b4-895d-3fa206648938-console-serving-cert\") pod \"d8771e5f-12f7-40b4-895d-3fa206648938\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") "
Sep 17 23:09:47.629658 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.629413    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-oauth-serving-cert\") pod \"d8771e5f-12f7-40b4-895d-3fa206648938\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") "
Sep 17 23:09:47.629658 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.629434    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8771e5f-12f7-40b4-895d-3fa206648938-console-oauth-config\") pod \"d8771e5f-12f7-40b4-895d-3fa206648938\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") "
Sep 17 23:09:47.629658 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.629624    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffpcx\" (UniqueName: \"kubernetes.io/projected/d8771e5f-12f7-40b4-895d-3fa206648938-kube-api-access-ffpcx\") pod \"d8771e5f-12f7-40b4-895d-3fa206648938\" (UID: \"d8771e5f-12f7-40b4-895d-3fa206648938\") "
Sep 17 23:09:47.629951 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.629859    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d8771e5f-12f7-40b4-895d-3fa206648938" (UID: "d8771e5f-12f7-40b4-895d-3fa206648938"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:09:47.630122 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.630096    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d8771e5f-12f7-40b4-895d-3fa206648938" (UID: "d8771e5f-12f7-40b4-895d-3fa206648938"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:09:47.630234 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.629954    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-service-ca" (OuterVolumeSpecName: "service-ca") pod "d8771e5f-12f7-40b4-895d-3fa206648938" (UID: "d8771e5f-12f7-40b4-895d-3fa206648938"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:09:47.630234 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.630063    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-console-config" (OuterVolumeSpecName: "console-config") pod "d8771e5f-12f7-40b4-895d-3fa206648938" (UID: "d8771e5f-12f7-40b4-895d-3fa206648938"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:09:47.631804 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.631785    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8771e5f-12f7-40b4-895d-3fa206648938-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d8771e5f-12f7-40b4-895d-3fa206648938" (UID: "d8771e5f-12f7-40b4-895d-3fa206648938"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:09:47.631804 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.631786    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8771e5f-12f7-40b4-895d-3fa206648938-kube-api-access-ffpcx" (OuterVolumeSpecName: "kube-api-access-ffpcx") pod "d8771e5f-12f7-40b4-895d-3fa206648938" (UID: "d8771e5f-12f7-40b4-895d-3fa206648938"). InnerVolumeSpecName "kube-api-access-ffpcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:09:47.631956 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.631850    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8771e5f-12f7-40b4-895d-3fa206648938-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d8771e5f-12f7-40b4-895d-3fa206648938" (UID: "d8771e5f-12f7-40b4-895d-3fa206648938"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:09:47.730615 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.730573    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffpcx\" (UniqueName: \"kubernetes.io/projected/d8771e5f-12f7-40b4-895d-3fa206648938-kube-api-access-ffpcx\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:09:47.730615 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.730607    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-trusted-ca-bundle\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:09:47.730615 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.730617    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-console-config\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:09:47.730615 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.730626    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:09:47.730914 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.730634    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8771e5f-12f7-40b4-895d-3fa206648938-console-serving-cert\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:09:47.730914 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.730643    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8771e5f-12f7-40b4-895d-3fa206648938-oauth-serving-cert\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:09:47.730914 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.730651    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8771e5f-12f7-40b4-895d-3fa206648938-console-oauth-config\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:09:47.883105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.883008    2568 generic.go:356] "Generic (PLEG): container finished" podID="d8771e5f-12f7-40b4-895d-3fa206648938" containerID="353874fb6d7a99f39c3abbaf075ae6c4e2873b0f24c2852e843d91b2a9e55293" exitCode=0
Sep 17 23:09:47.883105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.883057    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-695bd4fb54-cktj7"
Sep 17 23:09:47.883312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.883059    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695bd4fb54-cktj7" event={"ID":"d8771e5f-12f7-40b4-895d-3fa206648938","Type":"ContainerDied","Data":"353874fb6d7a99f39c3abbaf075ae6c4e2873b0f24c2852e843d91b2a9e55293"}
Sep 17 23:09:47.883312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.883199    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-695bd4fb54-cktj7" event={"ID":"d8771e5f-12f7-40b4-895d-3fa206648938","Type":"ContainerDied","Data":"2c39f21e03849c311afd72ea8928c5125aea6903a50f0926e10678216e9d60ef"}
Sep 17 23:09:47.883312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.883227    2568 scope.go:117] "RemoveContainer" containerID="353874fb6d7a99f39c3abbaf075ae6c4e2873b0f24c2852e843d91b2a9e55293"
Sep 17 23:09:47.893269 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.893168    2568 scope.go:117] "RemoveContainer" containerID="353874fb6d7a99f39c3abbaf075ae6c4e2873b0f24c2852e843d91b2a9e55293"
Sep 17 23:09:47.893590 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:09:47.893568    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353874fb6d7a99f39c3abbaf075ae6c4e2873b0f24c2852e843d91b2a9e55293\": container with ID starting with 353874fb6d7a99f39c3abbaf075ae6c4e2873b0f24c2852e843d91b2a9e55293 not found: ID does not exist" containerID="353874fb6d7a99f39c3abbaf075ae6c4e2873b0f24c2852e843d91b2a9e55293"
Sep 17 23:09:47.893669 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.893598    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353874fb6d7a99f39c3abbaf075ae6c4e2873b0f24c2852e843d91b2a9e55293"} err="failed to get container status \"353874fb6d7a99f39c3abbaf075ae6c4e2873b0f24c2852e843d91b2a9e55293\": rpc error: code = NotFound desc = could not find container \"353874fb6d7a99f39c3abbaf075ae6c4e2873b0f24c2852e843d91b2a9e55293\": container with ID starting with 353874fb6d7a99f39c3abbaf075ae6c4e2873b0f24c2852e843d91b2a9e55293 not found: ID does not exist"
Sep 17 23:09:47.905582 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.905544    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-695bd4fb54-cktj7"]
Sep 17 23:09:47.909085 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.909060    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-695bd4fb54-cktj7"]
Sep 17 23:09:47.988706 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:09:47.988673    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8771e5f-12f7-40b4-895d-3fa206648938" path="/var/lib/kubelet/pods/d8771e5f-12f7-40b4-895d-3fa206648938/volumes"
Sep 17 23:10:10.268109 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:10.268069    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:10.358425 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:10.358391    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:17.403507 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:17.403473    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:32.137057 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:32.137021    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:32.957020 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:32.956986    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:35.760479 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:35.760445    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:35.863218 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:35.863177    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:36.979682 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:36.979647    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:37.109006 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:37.108968    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:37.969774 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:37.969742    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:38.074324 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:38.074288    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:39.979149 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:39.979115    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:40.390242 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:40.390210    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:42.552669 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:42.552632    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:42.615849 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:42.615817    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:46.409249 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:46.409211    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:46.463205 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:46.463161    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:48.551474 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:48.551441    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:48.649840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:48.649803    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:50.156667 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.156636    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:50.228090 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.228060    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:50.775620 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.775585    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8w9cg"]
Sep 17 23:10:50.775925 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.775911    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf77948-c8eb-4f15-9021-5be425641cec" containerName="console"
Sep 17 23:10:50.775975 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.775927    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf77948-c8eb-4f15-9021-5be425641cec" containerName="console"
Sep 17 23:10:50.775975 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.775946    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8771e5f-12f7-40b4-895d-3fa206648938" containerName="console"
Sep 17 23:10:50.775975 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.775952    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8771e5f-12f7-40b4-895d-3fa206648938" containerName="console"
Sep 17 23:10:50.776063 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.776002    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf77948-c8eb-4f15-9021-5be425641cec" containerName="console"
Sep 17 23:10:50.776063 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.776009    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8771e5f-12f7-40b4-895d-3fa206648938" containerName="console"
Sep 17 23:10:50.778425 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.778409    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8w9cg"
Sep 17 23:10:50.780845 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.780822    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:50.781002 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.780925    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:50.781783 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.781759    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9dllg\""
Sep 17 23:10:50.787212 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.787189    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8w9cg"]
Sep 17 23:10:50.792694 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.792674    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07dcf854-06eb-4a91-8f2b-8cabdf11478e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8w9cg\" (UID: \"07dcf854-06eb-4a91-8f2b-8cabdf11478e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8w9cg"
Sep 17 23:10:50.792794 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.792755    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9rtc\" (UniqueName: \"kubernetes.io/projected/07dcf854-06eb-4a91-8f2b-8cabdf11478e-kube-api-access-t9rtc\") pod \"openshift-lws-operator-fd8ccff4c-8w9cg\" (UID: \"07dcf854-06eb-4a91-8f2b-8cabdf11478e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8w9cg"
Sep 17 23:10:50.894144 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.894108    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9rtc\" (UniqueName: \"kubernetes.io/projected/07dcf854-06eb-4a91-8f2b-8cabdf11478e-kube-api-access-t9rtc\") pod \"openshift-lws-operator-fd8ccff4c-8w9cg\" (UID: \"07dcf854-06eb-4a91-8f2b-8cabdf11478e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8w9cg"
Sep 17 23:10:50.894325 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.894159    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07dcf854-06eb-4a91-8f2b-8cabdf11478e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8w9cg\" (UID: \"07dcf854-06eb-4a91-8f2b-8cabdf11478e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8w9cg"
Sep 17 23:10:50.894492 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.894477    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07dcf854-06eb-4a91-8f2b-8cabdf11478e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8w9cg\" (UID: \"07dcf854-06eb-4a91-8f2b-8cabdf11478e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8w9cg"
Sep 17 23:10:50.902310 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:50.902285    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9rtc\" (UniqueName: \"kubernetes.io/projected/07dcf854-06eb-4a91-8f2b-8cabdf11478e-kube-api-access-t9rtc\") pod \"openshift-lws-operator-fd8ccff4c-8w9cg\" (UID: \"07dcf854-06eb-4a91-8f2b-8cabdf11478e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8w9cg"
Sep 17 23:10:51.095605 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:51.095498    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8w9cg"
Sep 17 23:10:51.362669 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:51.362636    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8w9cg"]
Sep 17 23:10:51.365268 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:10:51.365229    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07dcf854_06eb_4a91_8f2b_8cabdf11478e.slice/crio-bb8806745e3cd45d12aa61238a7e0930bd5bd8480fd0dff3509314d2f2e2906c WatchSource:0}: Error finding container bb8806745e3cd45d12aa61238a7e0930bd5bd8480fd0dff3509314d2f2e2906c: Status 404 returned error can't find the container with id bb8806745e3cd45d12aa61238a7e0930bd5bd8480fd0dff3509314d2f2e2906c
Sep 17 23:10:52.126236 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:52.126188    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8w9cg" event={"ID":"07dcf854-06eb-4a91-8f2b-8cabdf11478e","Type":"ContainerStarted","Data":"bb8806745e3cd45d12aa61238a7e0930bd5bd8480fd0dff3509314d2f2e2906c"}
Sep 17 23:10:54.136406 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:54.136365    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8w9cg" event={"ID":"07dcf854-06eb-4a91-8f2b-8cabdf11478e","Type":"ContainerStarted","Data":"16b81e5667e113233d5ea172173afe7290279c7cd8be99d412f568555578699a"}
Sep 17 23:10:54.157700 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:54.157649    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8w9cg" podStartSLOduration=1.76645739 podStartE2EDuration="4.157632646s" totalImagesPullingTime="2.391175256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:54.156477031 +0000 UTC m=+252.380152783" watchObservedRunningTime="2026-09-17 23:10:54.157632646 +0000 UTC m=+252.381308393"
Sep 17 23:10:54.876201 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:54.876166    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:54.977668 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:54.977632    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:56.868060 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:56.868029    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:58.395664 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:58.395576    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:58.493365 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:58.493329    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:10:58.510023 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:58.509992    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xrndf"]
Sep 17 23:10:58.512227 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:58.512211    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xrndf"
Sep 17 23:10:58.514503 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:58.514482    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-x4xpr\""
Sep 17 23:10:58.514604 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:58.514523    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 23:10:58.515242 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:58.515227    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 23:10:58.523032 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:58.523007    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xrndf"]
Sep 17 23:10:58.561244 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:58.561204    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl2zb\" (UniqueName: \"kubernetes.io/projected/784854ba-551c-4d3a-8316-d515e3c9548e-kube-api-access-rl2zb\") pod \"cert-manager-55b4d954d5-xrndf\" (UID: \"784854ba-551c-4d3a-8316-d515e3c9548e\") " pod="cert-manager/cert-manager-55b4d954d5-xrndf"
Sep 17 23:10:58.561436 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:58.561267    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/784854ba-551c-4d3a-8316-d515e3c9548e-bound-sa-token\") pod \"cert-manager-55b4d954d5-xrndf\" (UID: \"784854ba-551c-4d3a-8316-d515e3c9548e\") " pod="cert-manager/cert-manager-55b4d954d5-xrndf"
Sep 17 23:10:58.662108 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:58.662025    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl2zb\" (UniqueName: \"kubernetes.io/projected/784854ba-551c-4d3a-8316-d515e3c9548e-kube-api-access-rl2zb\") pod \"cert-manager-55b4d954d5-xrndf\" (UID: \"784854ba-551c-4d3a-8316-d515e3c9548e\") " pod="cert-manager/cert-manager-55b4d954d5-xrndf"
Sep 17 23:10:58.662108 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:58.662088    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/784854ba-551c-4d3a-8316-d515e3c9548e-bound-sa-token\") pod \"cert-manager-55b4d954d5-xrndf\" (UID: \"784854ba-551c-4d3a-8316-d515e3c9548e\") " pod="cert-manager/cert-manager-55b4d954d5-xrndf"
Sep 17 23:10:58.671694 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:58.671659    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/784854ba-551c-4d3a-8316-d515e3c9548e-bound-sa-token\") pod \"cert-manager-55b4d954d5-xrndf\" (UID: \"784854ba-551c-4d3a-8316-d515e3c9548e\") " pod="cert-manager/cert-manager-55b4d954d5-xrndf"
Sep 17 23:10:58.671835 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:58.671798    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl2zb\" (UniqueName: \"kubernetes.io/projected/784854ba-551c-4d3a-8316-d515e3c9548e-kube-api-access-rl2zb\") pod \"cert-manager-55b4d954d5-xrndf\" (UID: \"784854ba-551c-4d3a-8316-d515e3c9548e\") " pod="cert-manager/cert-manager-55b4d954d5-xrndf"
Sep 17 23:10:58.821103 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:58.821069    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xrndf"
Sep 17 23:10:59.090390 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:59.090089    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xrndf"]
Sep 17 23:10:59.092914 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:10:59.092886    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod784854ba_551c_4d3a_8316_d515e3c9548e.slice/crio-8f5f6060d573ebd4425908eeee520fac51512105ed179cb7fb02d29b1ebd4bc4 WatchSource:0}: Error finding container 8f5f6060d573ebd4425908eeee520fac51512105ed179cb7fb02d29b1ebd4bc4: Status 404 returned error can't find the container with id 8f5f6060d573ebd4425908eeee520fac51512105ed179cb7fb02d29b1ebd4bc4
Sep 17 23:10:59.154836 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:10:59.154803    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xrndf" event={"ID":"784854ba-551c-4d3a-8316-d515e3c9548e","Type":"ContainerStarted","Data":"8f5f6060d573ebd4425908eeee520fac51512105ed179cb7fb02d29b1ebd4bc4"}
Sep 17 23:11:02.170605 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:02.170570    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xrndf" event={"ID":"784854ba-551c-4d3a-8316-d515e3c9548e","Type":"ContainerStarted","Data":"44e46b71d1503ea3d5f47953207da831ec259f670267773e738daaa64c1c7c6e"}
Sep 17 23:11:02.188499 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:02.188253    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-xrndf" podStartSLOduration=1.225887269 podStartE2EDuration="4.188233192s" totalImagesPullingTime="2.962345923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:02.18799362 +0000 UTC m=+260.411669372" watchObservedRunningTime="2026-09-17 23:11:02.188233192 +0000 UTC m=+260.411908940"
Sep 17 23:11:05.091343 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.091303    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:05.143008 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.142972    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"]
Sep 17 23:11:05.145433 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.145412    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:05.149537 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.149514    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xfzjq\""
Sep 17 23:11:05.149681 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.149625    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 23:11:05.149681 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.149642    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 23:11:05.149795 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.149641    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 23:11:05.156133 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.156109    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"]
Sep 17 23:11:05.170412 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.170386    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:05.215366 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.215328    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ebdf634-ff58-4fba-8b41-03f09f36daf5-metrics-cert\") pod \"lws-controller-manager-858688bfcc-xs7c6\" (UID: \"5ebdf634-ff58-4fba-8b41-03f09f36daf5\") " pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:05.215366 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.215369    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ebdf634-ff58-4fba-8b41-03f09f36daf5-cert\") pod \"lws-controller-manager-858688bfcc-xs7c6\" (UID: \"5ebdf634-ff58-4fba-8b41-03f09f36daf5\") " pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:05.215567 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.215395    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5ebdf634-ff58-4fba-8b41-03f09f36daf5-manager-config\") pod \"lws-controller-manager-858688bfcc-xs7c6\" (UID: \"5ebdf634-ff58-4fba-8b41-03f09f36daf5\") " pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:05.215567 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.215494    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k54vk\" (UniqueName: \"kubernetes.io/projected/5ebdf634-ff58-4fba-8b41-03f09f36daf5-kube-api-access-k54vk\") pod \"lws-controller-manager-858688bfcc-xs7c6\" (UID: \"5ebdf634-ff58-4fba-8b41-03f09f36daf5\") " pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:05.316811 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.316773    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k54vk\" (UniqueName: \"kubernetes.io/projected/5ebdf634-ff58-4fba-8b41-03f09f36daf5-kube-api-access-k54vk\") pod \"lws-controller-manager-858688bfcc-xs7c6\" (UID: \"5ebdf634-ff58-4fba-8b41-03f09f36daf5\") " pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:05.316983 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.316836    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ebdf634-ff58-4fba-8b41-03f09f36daf5-metrics-cert\") pod \"lws-controller-manager-858688bfcc-xs7c6\" (UID: \"5ebdf634-ff58-4fba-8b41-03f09f36daf5\") " pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:05.316983 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.316887    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ebdf634-ff58-4fba-8b41-03f09f36daf5-cert\") pod \"lws-controller-manager-858688bfcc-xs7c6\" (UID: \"5ebdf634-ff58-4fba-8b41-03f09f36daf5\") " pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:05.316983 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.316921    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5ebdf634-ff58-4fba-8b41-03f09f36daf5-manager-config\") pod \"lws-controller-manager-858688bfcc-xs7c6\" (UID: \"5ebdf634-ff58-4fba-8b41-03f09f36daf5\") " pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:05.317690 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.317667    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5ebdf634-ff58-4fba-8b41-03f09f36daf5-manager-config\") pod \"lws-controller-manager-858688bfcc-xs7c6\" (UID: \"5ebdf634-ff58-4fba-8b41-03f09f36daf5\") " pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:05.319407 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.319389    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ebdf634-ff58-4fba-8b41-03f09f36daf5-cert\") pod \"lws-controller-manager-858688bfcc-xs7c6\" (UID: \"5ebdf634-ff58-4fba-8b41-03f09f36daf5\") " pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:05.319490 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.319432    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ebdf634-ff58-4fba-8b41-03f09f36daf5-metrics-cert\") pod \"lws-controller-manager-858688bfcc-xs7c6\" (UID: \"5ebdf634-ff58-4fba-8b41-03f09f36daf5\") " pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:05.327110 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.327090    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k54vk\" (UniqueName: \"kubernetes.io/projected/5ebdf634-ff58-4fba-8b41-03f09f36daf5-kube-api-access-k54vk\") pod \"lws-controller-manager-858688bfcc-xs7c6\" (UID: \"5ebdf634-ff58-4fba-8b41-03f09f36daf5\") " pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:05.455333 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.455238    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:05.737134 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:05.737063    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"]
Sep 17 23:11:05.740013 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:11:05.739981    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebdf634_ff58_4fba_8b41_03f09f36daf5.slice/crio-0fe557871b8fdac299b62434d8370a0ee34fa55a0e0f42e299bd8afb6c4e056d WatchSource:0}: Error finding container 0fe557871b8fdac299b62434d8370a0ee34fa55a0e0f42e299bd8afb6c4e056d: Status 404 returned error can't find the container with id 0fe557871b8fdac299b62434d8370a0ee34fa55a0e0f42e299bd8afb6c4e056d
Sep 17 23:11:06.186850 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:06.186818    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6" event={"ID":"5ebdf634-ff58-4fba-8b41-03f09f36daf5","Type":"ContainerStarted","Data":"0fe557871b8fdac299b62434d8370a0ee34fa55a0e0f42e299bd8afb6c4e056d"}
Sep 17 23:11:08.197945 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:08.197906    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6" event={"ID":"5ebdf634-ff58-4fba-8b41-03f09f36daf5","Type":"ContainerStarted","Data":"2e0245230f4fece3f2fdcaedb359e158d831d14ed1cc6c71bbd56394dcab638e"}
Sep 17 23:11:08.198433 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:08.198044    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:08.226226 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:08.226179    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6" podStartSLOduration=1.6773442109999999 podStartE2EDuration="3.226161279s" totalImagesPullingTime="1.548817068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:08.225585701 +0000 UTC m=+266.449261449" watchObservedRunningTime="2026-09-17 23:11:08.226161279 +0000 UTC m=+266.449837027"
Sep 17 23:11:08.368244 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:08.368189    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:08.469222 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:08.469142    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:08.967122 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:08.967091    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:09.159600 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.159569    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:09.263311 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.263224    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:09.296271 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.296241    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"]
Sep 17 23:11:09.298708 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.298689    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.301336 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.301311    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 23:11:09.301456 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.301411    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 23:11:09.301926 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.301901    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 23:11:09.302041 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.301970    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-czjhc\""
Sep 17 23:11:09.302210 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.302191    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 23:11:09.316807 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.316781    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"]
Sep 17 23:11:09.359433 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.359396    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10f49254-7996-4cbf-859a-cb932d1ef50b-webhook-cert\") pod \"opendatahub-operator-controller-manager-697bb5576d-484bm\" (UID: \"10f49254-7996-4cbf-859a-cb932d1ef50b\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.359628 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.359462    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f49254-7996-4cbf-859a-cb932d1ef50b-tmp\") pod \"opendatahub-operator-controller-manager-697bb5576d-484bm\" (UID: \"10f49254-7996-4cbf-859a-cb932d1ef50b\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.359628 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.359532    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/10f49254-7996-4cbf-859a-cb932d1ef50b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-697bb5576d-484bm\" (UID: \"10f49254-7996-4cbf-859a-cb932d1ef50b\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.359628 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.359578    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzgrm\" (UniqueName: \"kubernetes.io/projected/10f49254-7996-4cbf-859a-cb932d1ef50b-kube-api-access-dzgrm\") pod \"opendatahub-operator-controller-manager-697bb5576d-484bm\" (UID: \"10f49254-7996-4cbf-859a-cb932d1ef50b\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.359628 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.359601    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/10f49254-7996-4cbf-859a-cb932d1ef50b-manifests\") pod \"opendatahub-operator-controller-manager-697bb5576d-484bm\" (UID: \"10f49254-7996-4cbf-859a-cb932d1ef50b\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.427501 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.427468    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:09.460492 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.460456    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10f49254-7996-4cbf-859a-cb932d1ef50b-webhook-cert\") pod \"opendatahub-operator-controller-manager-697bb5576d-484bm\" (UID: \"10f49254-7996-4cbf-859a-cb932d1ef50b\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.460652 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.460507    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f49254-7996-4cbf-859a-cb932d1ef50b-tmp\") pod \"opendatahub-operator-controller-manager-697bb5576d-484bm\" (UID: \"10f49254-7996-4cbf-859a-cb932d1ef50b\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.460652 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.460547    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/10f49254-7996-4cbf-859a-cb932d1ef50b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-697bb5576d-484bm\" (UID: \"10f49254-7996-4cbf-859a-cb932d1ef50b\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.460652 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.460597    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzgrm\" (UniqueName: \"kubernetes.io/projected/10f49254-7996-4cbf-859a-cb932d1ef50b-kube-api-access-dzgrm\") pod \"opendatahub-operator-controller-manager-697bb5576d-484bm\" (UID: \"10f49254-7996-4cbf-859a-cb932d1ef50b\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.460652 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.460627    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/10f49254-7996-4cbf-859a-cb932d1ef50b-manifests\") pod \"opendatahub-operator-controller-manager-697bb5576d-484bm\" (UID: \"10f49254-7996-4cbf-859a-cb932d1ef50b\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.461036 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.461005    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/10f49254-7996-4cbf-859a-cb932d1ef50b-manifests\") pod \"opendatahub-operator-controller-manager-697bb5576d-484bm\" (UID: \"10f49254-7996-4cbf-859a-cb932d1ef50b\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.462973 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.462945    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10f49254-7996-4cbf-859a-cb932d1ef50b-tmp\") pod \"opendatahub-operator-controller-manager-697bb5576d-484bm\" (UID: \"10f49254-7996-4cbf-859a-cb932d1ef50b\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.463067 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.463046    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10f49254-7996-4cbf-859a-cb932d1ef50b-webhook-cert\") pod \"opendatahub-operator-controller-manager-697bb5576d-484bm\" (UID: \"10f49254-7996-4cbf-859a-cb932d1ef50b\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.463172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.463154    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/10f49254-7996-4cbf-859a-cb932d1ef50b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-697bb5576d-484bm\" (UID: \"10f49254-7996-4cbf-859a-cb932d1ef50b\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.468395 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.468377    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzgrm\" (UniqueName: \"kubernetes.io/projected/10f49254-7996-4cbf-859a-cb932d1ef50b-kube-api-access-dzgrm\") pod \"opendatahub-operator-controller-manager-697bb5576d-484bm\" (UID: \"10f49254-7996-4cbf-859a-cb932d1ef50b\") " pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.526735 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.526709    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:09.613470 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.613438    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:09.623575 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.623546    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:09.753249 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.753219    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:09.855706 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.855605    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:09.893987 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.893963    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"]
Sep 17 23:11:09.896141 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:11:09.896112    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f49254_7996_4cbf_859a_cb932d1ef50b.slice/crio-7c4aeb317c33551835b3e9004f72a738509f24f655bfedacdb44615b1219cf74 WatchSource:0}: Error finding container 7c4aeb317c33551835b3e9004f72a738509f24f655bfedacdb44615b1219cf74: Status 404 returned error can't find the container with id 7c4aeb317c33551835b3e9004f72a738509f24f655bfedacdb44615b1219cf74
Sep 17 23:11:09.958927 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:09.958891    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:10.206927 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:10.206808    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm" event={"ID":"10f49254-7996-4cbf-859a-cb932d1ef50b","Type":"ContainerStarted","Data":"7c4aeb317c33551835b3e9004f72a738509f24f655bfedacdb44615b1219cf74"}
Sep 17 23:11:13.222625 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:13.222585    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm" event={"ID":"10f49254-7996-4cbf-859a-cb932d1ef50b","Type":"ContainerStarted","Data":"f6e9bdac77c5dd57896e45a1b768fb9758c14e7ce5372b055df4bbbc2f7400e1"}
Sep 17 23:11:14.227975 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:14.227888    2568 generic.go:356] "Generic (PLEG): container finished" podID="10f49254-7996-4cbf-859a-cb932d1ef50b" containerID="f6e9bdac77c5dd57896e45a1b768fb9758c14e7ce5372b055df4bbbc2f7400e1" exitCode=0
Sep 17 23:11:14.228358 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:14.227964    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm" event={"ID":"10f49254-7996-4cbf-859a-cb932d1ef50b","Type":"ContainerDied","Data":"f6e9bdac77c5dd57896e45a1b768fb9758c14e7ce5372b055df4bbbc2f7400e1"}
Sep 17 23:11:15.234271 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:15.234243    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm" event={"ID":"10f49254-7996-4cbf-859a-cb932d1ef50b","Type":"ContainerStarted","Data":"b5b8a7f0e88075213da2460d3823ff08735957847da000ce02f1e5fffaca316e"}
Sep 17 23:11:15.234662 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:15.234384    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:15.265704 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:15.265652    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm" podStartSLOduration=2.092882762 podStartE2EDuration="6.265633463s" totalImagesPullingTime="3.084314863s" totalInitContainerRuntime="1.088435838s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:15.264774458 +0000 UTC m=+273.488450219" watchObservedRunningTime="2026-09-17 23:11:15.265633463 +0000 UTC m=+273.489309211"
Sep 17 23:11:16.330104 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:16.330071    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:16.430989 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:16.430947    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:19.203488 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:19.203457    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-858688bfcc-xs7c6"
Sep 17 23:11:26.240471 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:26.240436    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-697bb5576d-484bm"
Sep 17 23:11:28.127187 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.127144    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:28.230841 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.230812    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:28.519458 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.519360    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:28.572837 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.572803    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x"]
Sep 17 23:11:28.637995 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.637957    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x"]
Sep 17 23:11:28.638159 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.638073    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x"
Sep 17 23:11:28.641912 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.641887    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rqpz6\""
Sep 17 23:11:28.641912 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.641895    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 17 23:11:28.642095 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.641951    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 23:11:28.642095 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.641956    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 23:11:28.642095 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.641968    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 17 23:11:28.655806 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.655784    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:28.724514 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.724475    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4r4x\" (UniqueName: \"kubernetes.io/projected/61fdd960-430c-40fa-8b74-e3bcd05b12c3-kube-api-access-t4r4x\") pod \"kube-auth-proxy-65f4688f79-k7p9x\" (UID: \"61fdd960-430c-40fa-8b74-e3bcd05b12c3\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x"
Sep 17 23:11:28.724514 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.724519    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61fdd960-430c-40fa-8b74-e3bcd05b12c3-tmp\") pod \"kube-auth-proxy-65f4688f79-k7p9x\" (UID: \"61fdd960-430c-40fa-8b74-e3bcd05b12c3\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x"
Sep 17 23:11:28.724716 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.724563    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fdd960-430c-40fa-8b74-e3bcd05b12c3-tls-certs\") pod \"kube-auth-proxy-65f4688f79-k7p9x\" (UID: \"61fdd960-430c-40fa-8b74-e3bcd05b12c3\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x"
Sep 17 23:11:28.824981 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.824950    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4r4x\" (UniqueName: \"kubernetes.io/projected/61fdd960-430c-40fa-8b74-e3bcd05b12c3-kube-api-access-t4r4x\") pod \"kube-auth-proxy-65f4688f79-k7p9x\" (UID: \"61fdd960-430c-40fa-8b74-e3bcd05b12c3\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x"
Sep 17 23:11:28.825146 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.824990    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61fdd960-430c-40fa-8b74-e3bcd05b12c3-tmp\") pod \"kube-auth-proxy-65f4688f79-k7p9x\" (UID: \"61fdd960-430c-40fa-8b74-e3bcd05b12c3\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x"
Sep 17 23:11:28.825146 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.825033    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fdd960-430c-40fa-8b74-e3bcd05b12c3-tls-certs\") pod \"kube-auth-proxy-65f4688f79-k7p9x\" (UID: \"61fdd960-430c-40fa-8b74-e3bcd05b12c3\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x"
Sep 17 23:11:28.827254 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.827230    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61fdd960-430c-40fa-8b74-e3bcd05b12c3-tmp\") pod \"kube-auth-proxy-65f4688f79-k7p9x\" (UID: \"61fdd960-430c-40fa-8b74-e3bcd05b12c3\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x"
Sep 17 23:11:28.827491 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.827475    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61fdd960-430c-40fa-8b74-e3bcd05b12c3-tls-certs\") pod \"kube-auth-proxy-65f4688f79-k7p9x\" (UID: \"61fdd960-430c-40fa-8b74-e3bcd05b12c3\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x"
Sep 17 23:11:28.834807 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.834789    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4r4x\" (UniqueName: \"kubernetes.io/projected/61fdd960-430c-40fa-8b74-e3bcd05b12c3-kube-api-access-t4r4x\") pod \"kube-auth-proxy-65f4688f79-k7p9x\" (UID: \"61fdd960-430c-40fa-8b74-e3bcd05b12c3\") " pod="openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x"
Sep 17 23:11:28.947442 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:28.947409    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x"
Sep 17 23:11:29.263498 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:29.263475    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x"]
Sep 17 23:11:29.265530 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:11:29.265499    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fdd960_430c_40fa_8b74_e3bcd05b12c3.slice/crio-7acf046eb16c74e023d4d5ea1b3e9c9a242251c42d98ff9b20c6d2e8a3eea95c WatchSource:0}: Error finding container 7acf046eb16c74e023d4d5ea1b3e9c9a242251c42d98ff9b20c6d2e8a3eea95c: Status 404 returned error can't find the container with id 7acf046eb16c74e023d4d5ea1b3e9c9a242251c42d98ff9b20c6d2e8a3eea95c
Sep 17 23:11:29.294403 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:29.294366    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x" event={"ID":"61fdd960-430c-40fa-8b74-e3bcd05b12c3","Type":"ContainerStarted","Data":"7acf046eb16c74e023d4d5ea1b3e9c9a242251c42d98ff9b20c6d2e8a3eea95c"}
Sep 17 23:11:30.022691 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:30.022613    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:30.107499 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:30.107430    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:31.086670 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:31.086623    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:33.318661 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:33.318630    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x" event={"ID":"61fdd960-430c-40fa-8b74-e3bcd05b12c3","Type":"ContainerStarted","Data":"f42e24ad99aa6683ac68507e401db0f32c894aecceffc466b74052706d1e77c4"}
Sep 17 23:11:33.340224 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:33.340174    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65f4688f79-k7p9x" podStartSLOduration=2.267252598 podStartE2EDuration="5.340158438s" totalImagesPullingTime="3.07290584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:33.33901901 +0000 UTC m=+291.562694761" watchObservedRunningTime="2026-09-17 23:11:33.340158438 +0000 UTC m=+291.563834185"
Sep 17 23:11:34.420179 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:34.420147    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:34.521582 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:34.521551    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:35.583195 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:35.583162    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:35.770147 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:35.770118    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:35.895799 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:35.895709    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:38.054292 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:38.054258    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:38.118642 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:38.118610    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:39.147568 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:39.147537    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:39.261393 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:39.261362    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:39.410607 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:39.410526    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:39.749990 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:39.749898    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:39.949237 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:39.949200    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:41.927655 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:41.927618    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:41.983218 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:41.983168    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:43.870969 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:43.870942    2568 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 23:11:48.791323 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:48.791283    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:48.878103 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:48.878065    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:48.948592 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:48.948560    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:49.081201 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:49.081110    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:50.103849 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:50.103807    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:50.190187 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:50.190154    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:50.248321 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:11:50.248289    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:28.740888 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.740838    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f8d85b594-78bzs"]
Sep 17 23:12:28.743182 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.743164    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.745719 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.745689    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rftn6\""
Sep 17 23:12:28.745970 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.745953    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 23:12:28.746751 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.746718    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 23:12:28.746825 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.746775    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 23:12:28.746825 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.746795    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 23:12:28.746825 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.746818    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 23:12:28.747111 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.746935    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 23:12:28.747111 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.746940    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 23:12:28.752904 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.752859    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 23:12:28.757247 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.757224    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f8d85b594-78bzs"]
Sep 17 23:12:28.825264 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.825234    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81c6cd29-1502-446f-8ac3-4f20bebed00b-console-oauth-config\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.825264 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.825266    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81c6cd29-1502-446f-8ac3-4f20bebed00b-service-ca\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.825484 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.825293    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81c6cd29-1502-446f-8ac3-4f20bebed00b-oauth-serving-cert\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.825484 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.825372    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c6cd29-1502-446f-8ac3-4f20bebed00b-trusted-ca-bundle\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.825484 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.825415    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81c6cd29-1502-446f-8ac3-4f20bebed00b-console-config\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.825484 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.825433    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brg4w\" (UniqueName: \"kubernetes.io/projected/81c6cd29-1502-446f-8ac3-4f20bebed00b-kube-api-access-brg4w\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.825484 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.825483    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81c6cd29-1502-446f-8ac3-4f20bebed00b-console-serving-cert\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.878313 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.878282    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:28.926532 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.926490    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81c6cd29-1502-446f-8ac3-4f20bebed00b-console-config\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.926532 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.926536    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brg4w\" (UniqueName: \"kubernetes.io/projected/81c6cd29-1502-446f-8ac3-4f20bebed00b-kube-api-access-brg4w\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.926803 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.926578    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81c6cd29-1502-446f-8ac3-4f20bebed00b-console-serving-cert\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.926803 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.926658    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81c6cd29-1502-446f-8ac3-4f20bebed00b-console-oauth-config\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.926803 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.926681    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81c6cd29-1502-446f-8ac3-4f20bebed00b-service-ca\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.926803 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.926707    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81c6cd29-1502-446f-8ac3-4f20bebed00b-oauth-serving-cert\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.926803 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.926743    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c6cd29-1502-446f-8ac3-4f20bebed00b-trusted-ca-bundle\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.927397 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.927369    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81c6cd29-1502-446f-8ac3-4f20bebed00b-console-config\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.927500 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.927370    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81c6cd29-1502-446f-8ac3-4f20bebed00b-oauth-serving-cert\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.927862 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.927839    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81c6cd29-1502-446f-8ac3-4f20bebed00b-trusted-ca-bundle\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.927957 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.927938    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81c6cd29-1502-446f-8ac3-4f20bebed00b-service-ca\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.929217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.929193    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81c6cd29-1502-446f-8ac3-4f20bebed00b-console-serving-cert\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.929217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.929207    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81c6cd29-1502-446f-8ac3-4f20bebed00b-console-oauth-config\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.939358 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.939332    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brg4w\" (UniqueName: \"kubernetes.io/projected/81c6cd29-1502-446f-8ac3-4f20bebed00b-kube-api-access-brg4w\") pod \"console-5f8d85b594-78bzs\" (UID: \"81c6cd29-1502-446f-8ac3-4f20bebed00b\") " pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:28.957306 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:28.957275    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:29.052858 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:29.052829    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:29.323556 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:29.323454    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f8d85b594-78bzs"]
Sep 17 23:12:29.326396 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:12:29.326362    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c6cd29_1502_446f_8ac3_4f20bebed00b.slice/crio-679535a9e10c28d3756fcaa78c94a045a7747c1f37d959e7765aaa4019ebf9ec WatchSource:0}: Error finding container 679535a9e10c28d3756fcaa78c94a045a7747c1f37d959e7765aaa4019ebf9ec: Status 404 returned error can't find the container with id 679535a9e10c28d3756fcaa78c94a045a7747c1f37d959e7765aaa4019ebf9ec
Sep 17 23:12:29.555936 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:29.555897    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8d85b594-78bzs" event={"ID":"81c6cd29-1502-446f-8ac3-4f20bebed00b","Type":"ContainerStarted","Data":"eeccb7fdb97b443845e282d4448d500a305c3a8196c104a549e6c81d5f949911"}
Sep 17 23:12:29.556105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:29.555949    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8d85b594-78bzs" event={"ID":"81c6cd29-1502-446f-8ac3-4f20bebed00b","Type":"ContainerStarted","Data":"679535a9e10c28d3756fcaa78c94a045a7747c1f37d959e7765aaa4019ebf9ec"}
Sep 17 23:12:29.577174 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:29.577080    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f8d85b594-78bzs" podStartSLOduration=1.5770632020000002 podStartE2EDuration="1.577063202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:29.575862889 +0000 UTC m=+347.799538643" watchObservedRunningTime="2026-09-17 23:12:29.577063202 +0000 UTC m=+347.800738950"
Sep 17 23:12:39.053267 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:39.053233    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:39.053726 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:39.053333    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:39.058122 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:39.058100    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:39.601647 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:39.601616    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f8d85b594-78bzs"
Sep 17 23:12:39.773564 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:39.773534    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:40.309037 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:40.308995    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:40.401622 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:40.401590    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:55.225619 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:55.225583    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:58.020081 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:58.020048    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:59.601828 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:12:59.601796    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:00.065415 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:00.065384    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:00.268282 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:00.268249    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:01.739666 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:01.739630    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:02.393553 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:02.393512    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:02.659904 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:02.659808    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:02.802639 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:02.802610    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:02.902796 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:02.902768    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:03.218984 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:03.218946    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:03.340861 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:03.340828    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:04.123730 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:04.123693    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:07.639322 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:07.639263    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:07.744803 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:07.744774    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:10.820408 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:10.820378    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:10.921390 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:10.921359    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:13.297049 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.297012    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:13.301529 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.301497    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc"]
Sep 17 23:13:13.304438 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.304419    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc"
Sep 17 23:13:13.306851 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.306829    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 23:13:13.307000 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.306858    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 23:13:13.307169 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.307152    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 23:13:13.307846 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.307827    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 23:13:13.307943 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.307855    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cqh4k\""
Sep 17 23:13:13.313138 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.313104    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc"]
Sep 17 23:13:13.319303 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.319279    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmlmx\" (UniqueName: \"kubernetes.io/projected/405b6ac1-52ac-40c2-90d3-1117943ebb71-kube-api-access-bmlmx\") pod \"kuadrant-console-plugin-fbdd4f445-9rnfc\" (UID: \"405b6ac1-52ac-40c2-90d3-1117943ebb71\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc"
Sep 17 23:13:13.319406 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.319324    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/405b6ac1-52ac-40c2-90d3-1117943ebb71-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9rnfc\" (UID: \"405b6ac1-52ac-40c2-90d3-1117943ebb71\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc"
Sep 17 23:13:13.319406 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.319353    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/405b6ac1-52ac-40c2-90d3-1117943ebb71-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9rnfc\" (UID: \"405b6ac1-52ac-40c2-90d3-1117943ebb71\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc"
Sep 17 23:13:13.420559 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.420530    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/405b6ac1-52ac-40c2-90d3-1117943ebb71-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9rnfc\" (UID: \"405b6ac1-52ac-40c2-90d3-1117943ebb71\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc"
Sep 17 23:13:13.420734 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.420596    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmlmx\" (UniqueName: \"kubernetes.io/projected/405b6ac1-52ac-40c2-90d3-1117943ebb71-kube-api-access-bmlmx\") pod \"kuadrant-console-plugin-fbdd4f445-9rnfc\" (UID: \"405b6ac1-52ac-40c2-90d3-1117943ebb71\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc"
Sep 17 23:13:13.420734 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.420635    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/405b6ac1-52ac-40c2-90d3-1117943ebb71-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9rnfc\" (UID: \"405b6ac1-52ac-40c2-90d3-1117943ebb71\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc"
Sep 17 23:13:13.421245 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.421227    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/405b6ac1-52ac-40c2-90d3-1117943ebb71-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9rnfc\" (UID: \"405b6ac1-52ac-40c2-90d3-1117943ebb71\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc"
Sep 17 23:13:13.423101 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.423079    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/405b6ac1-52ac-40c2-90d3-1117943ebb71-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9rnfc\" (UID: \"405b6ac1-52ac-40c2-90d3-1117943ebb71\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc"
Sep 17 23:13:13.429986 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.429962    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmlmx\" (UniqueName: \"kubernetes.io/projected/405b6ac1-52ac-40c2-90d3-1117943ebb71-kube-api-access-bmlmx\") pod \"kuadrant-console-plugin-fbdd4f445-9rnfc\" (UID: \"405b6ac1-52ac-40c2-90d3-1117943ebb71\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc"
Sep 17 23:13:13.449372 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.449336    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:13.614552 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.614455    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc"
Sep 17 23:13:13.892839 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.892760    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc"]
Sep 17 23:13:13.896230 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:13:13.896205    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod405b6ac1_52ac_40c2_90d3_1117943ebb71.slice/crio-3011cd1a0a16e3239a3f34cc759f649e52010c00691521a00504907084e67b41 WatchSource:0}: Error finding container 3011cd1a0a16e3239a3f34cc759f649e52010c00691521a00504907084e67b41: Status 404 returned error can't find the container with id 3011cd1a0a16e3239a3f34cc759f649e52010c00691521a00504907084e67b41
Sep 17 23:13:13.897685 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:13.897665    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:13:14.757661 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:14.757621    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc" event={"ID":"405b6ac1-52ac-40c2-90d3-1117943ebb71","Type":"ContainerStarted","Data":"3011cd1a0a16e3239a3f34cc759f649e52010c00691521a00504907084e67b41"}
Sep 17 23:13:15.292157 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:15.292120    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:15.394461 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:15.394421    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:18.776543 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:18.776507    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc" event={"ID":"405b6ac1-52ac-40c2-90d3-1117943ebb71","Type":"ContainerStarted","Data":"3e10a26ae97b4e4336987fcfe62f63464e69079503f16206cda9b6d5f1313d68"}
Sep 17 23:13:18.798798 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:18.798745    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9rnfc" podStartSLOduration=1.660116513 podStartE2EDuration="5.798729124s" totalImagesPullingTime="4.138612611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:18.797498707 +0000 UTC m=+397.021174456" watchObservedRunningTime="2026-09-17 23:13:18.798729124 +0000 UTC m=+397.022404872"
Sep 17 23:13:20.028560 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:20.028527    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:20.125720 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:20.125685    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:20.226537 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:20.226502    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:23.503600 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:23.503569    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"]
Sep 17 23:13:23.505845 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:23.505825    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"
Sep 17 23:13:23.508418 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:23.508401    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sjqbx\""
Sep 17 23:13:23.522361 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:23.522330    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"]
Sep 17 23:13:23.612384 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:23.612349    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ec4c232-7f70-4a98-a576-bb87a43cefd6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-gk89g\" (UID: \"6ec4c232-7f70-4a98-a576-bb87a43cefd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"
Sep 17 23:13:23.612570 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:23.612447    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl9jm\" (UniqueName: \"kubernetes.io/projected/6ec4c232-7f70-4a98-a576-bb87a43cefd6-kube-api-access-sl9jm\") pod \"kuadrant-operator-controller-manager-5f685c496f-gk89g\" (UID: \"6ec4c232-7f70-4a98-a576-bb87a43cefd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"
Sep 17 23:13:23.712811 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:23.712775    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl9jm\" (UniqueName: \"kubernetes.io/projected/6ec4c232-7f70-4a98-a576-bb87a43cefd6-kube-api-access-sl9jm\") pod \"kuadrant-operator-controller-manager-5f685c496f-gk89g\" (UID: \"6ec4c232-7f70-4a98-a576-bb87a43cefd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"
Sep 17 23:13:23.713012 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:23.712832    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ec4c232-7f70-4a98-a576-bb87a43cefd6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-gk89g\" (UID: \"6ec4c232-7f70-4a98-a576-bb87a43cefd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"
Sep 17 23:13:23.713214 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:23.713194    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ec4c232-7f70-4a98-a576-bb87a43cefd6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-gk89g\" (UID: \"6ec4c232-7f70-4a98-a576-bb87a43cefd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"
Sep 17 23:13:23.729779 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:23.729743    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl9jm\" (UniqueName: \"kubernetes.io/projected/6ec4c232-7f70-4a98-a576-bb87a43cefd6-kube-api-access-sl9jm\") pod \"kuadrant-operator-controller-manager-5f685c496f-gk89g\" (UID: \"6ec4c232-7f70-4a98-a576-bb87a43cefd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"
Sep 17 23:13:23.815711 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:23.815672    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"
Sep 17 23:13:24.095032 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:24.094998    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"]
Sep 17 23:13:24.509883 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:24.509775    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"]
Sep 17 23:13:24.536237 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:24.536159    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"]
Sep 17 23:13:26.694708 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:26.694665    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:26.808424 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:26.808383    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:27.821356 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:27.821322    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g" podUID="6ec4c232-7f70-4a98-a576-bb87a43cefd6" containerName="manager" containerID="cri-o://bb93092feb747ca9b8216d76480bfe8c9d36868a4d081c6691868c8adff929fc" gracePeriod=2
Sep 17 23:13:29.888814 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:29.888727    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:29.978237 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:29.978215    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"
Sep 17 23:13:29.988603 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:29.988576    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:30.073298 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:30.073263    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ec4c232-7f70-4a98-a576-bb87a43cefd6-extensions-socket-volume\") pod \"6ec4c232-7f70-4a98-a576-bb87a43cefd6\" (UID: \"6ec4c232-7f70-4a98-a576-bb87a43cefd6\") "
Sep 17 23:13:30.073460 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:30.073317    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl9jm\" (UniqueName: \"kubernetes.io/projected/6ec4c232-7f70-4a98-a576-bb87a43cefd6-kube-api-access-sl9jm\") pod \"6ec4c232-7f70-4a98-a576-bb87a43cefd6\" (UID: \"6ec4c232-7f70-4a98-a576-bb87a43cefd6\") "
Sep 17 23:13:30.073576 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:30.073552    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec4c232-7f70-4a98-a576-bb87a43cefd6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6ec4c232-7f70-4a98-a576-bb87a43cefd6" (UID: "6ec4c232-7f70-4a98-a576-bb87a43cefd6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:13:30.075524 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:30.075454    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec4c232-7f70-4a98-a576-bb87a43cefd6-kube-api-access-sl9jm" (OuterVolumeSpecName: "kube-api-access-sl9jm") pod "6ec4c232-7f70-4a98-a576-bb87a43cefd6" (UID: "6ec4c232-7f70-4a98-a576-bb87a43cefd6"). InnerVolumeSpecName "kube-api-access-sl9jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:13:30.174382 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:30.174345    2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ec4c232-7f70-4a98-a576-bb87a43cefd6-extensions-socket-volume\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:13:30.174382 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:30.174377    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl9jm\" (UniqueName: \"kubernetes.io/projected/6ec4c232-7f70-4a98-a576-bb87a43cefd6-kube-api-access-sl9jm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:13:30.837717 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:30.837680    2568 generic.go:356] "Generic (PLEG): container finished" podID="6ec4c232-7f70-4a98-a576-bb87a43cefd6" containerID="bb93092feb747ca9b8216d76480bfe8c9d36868a4d081c6691868c8adff929fc" exitCode=137
Sep 17 23:13:30.838001 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:30.837730    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"
Sep 17 23:13:30.838001 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:30.837777    2568 scope.go:117] "RemoveContainer" containerID="bb93092feb747ca9b8216d76480bfe8c9d36868a4d081c6691868c8adff929fc"
Sep 17 23:13:30.846003 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:30.845985    2568 scope.go:117] "RemoveContainer" containerID="bb93092feb747ca9b8216d76480bfe8c9d36868a4d081c6691868c8adff929fc"
Sep 17 23:13:30.846305 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:13:30.846284    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb93092feb747ca9b8216d76480bfe8c9d36868a4d081c6691868c8adff929fc\": container with ID starting with bb93092feb747ca9b8216d76480bfe8c9d36868a4d081c6691868c8adff929fc not found: ID does not exist" containerID="bb93092feb747ca9b8216d76480bfe8c9d36868a4d081c6691868c8adff929fc"
Sep 17 23:13:30.846365 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:30.846314    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb93092feb747ca9b8216d76480bfe8c9d36868a4d081c6691868c8adff929fc"} err="failed to get container status \"bb93092feb747ca9b8216d76480bfe8c9d36868a4d081c6691868c8adff929fc\": rpc error: code = NotFound desc = could not find container \"bb93092feb747ca9b8216d76480bfe8c9d36868a4d081c6691868c8adff929fc\": container with ID starting with bb93092feb747ca9b8216d76480bfe8c9d36868a4d081c6691868c8adff929fc not found: ID does not exist"
Sep 17 23:13:30.847938 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:13:30.847915    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-gk89g\" is forbidden: User \"system:node:ip-10-0-140-44.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-44.ec2.internal' and this object" podUID="6ec4c232-7f70-4a98-a576-bb87a43cefd6" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gk89g"
Sep 17 23:13:31.989792 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:31.989760    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ec4c232-7f70-4a98-a576-bb87a43cefd6" path="/var/lib/kubelet/pods/6ec4c232-7f70-4a98-a576-bb87a43cefd6/volumes"
Sep 17 23:13:34.314045 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:34.314012    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:34.374060 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:34.374016    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:34.484856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:34.484818    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:34.586594 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:34.586525    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:39.508457 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:39.508421    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:39.609006 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:39.608976    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:39.706631 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:39.706592    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:52.841770 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:52.841737    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:52.985494 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:13:52.985460    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:02.308489 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:02.308401    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:07.634809 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.634771    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7ddsw"]
Sep 17 23:14:07.635319 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.635300    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec4c232-7f70-4a98-a576-bb87a43cefd6" containerName="manager"
Sep 17 23:14:07.635392 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.635322    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec4c232-7f70-4a98-a576-bb87a43cefd6" containerName="manager"
Sep 17 23:14:07.635447 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.635404    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ec4c232-7f70-4a98-a576-bb87a43cefd6" containerName="manager"
Sep 17 23:14:07.637558 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.637539    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7ddsw"
Sep 17 23:14:07.644698 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.644678    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 23:14:07.663613 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.663578    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7ddsw"]
Sep 17 23:14:07.668509 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.668481    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:07.694919 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.694856    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5136dce1-1b36-4376-86f0-2d3f8638b07b-config-file\") pod \"limitador-limitador-8588749977-7ddsw\" (UID: \"5136dce1-1b36-4376-86f0-2d3f8638b07b\") " pod="kuadrant-system/limitador-limitador-8588749977-7ddsw"
Sep 17 23:14:07.695126 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.694973    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5d8b\" (UniqueName: \"kubernetes.io/projected/5136dce1-1b36-4376-86f0-2d3f8638b07b-kube-api-access-j5d8b\") pod \"limitador-limitador-8588749977-7ddsw\" (UID: \"5136dce1-1b36-4376-86f0-2d3f8638b07b\") " pod="kuadrant-system/limitador-limitador-8588749977-7ddsw"
Sep 17 23:14:07.731234 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.731200    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7ddsw"]
Sep 17 23:14:07.769458 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.769425    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:07.795796 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.795760    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5136dce1-1b36-4376-86f0-2d3f8638b07b-config-file\") pod \"limitador-limitador-8588749977-7ddsw\" (UID: \"5136dce1-1b36-4376-86f0-2d3f8638b07b\") " pod="kuadrant-system/limitador-limitador-8588749977-7ddsw"
Sep 17 23:14:07.796001 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.795823    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5d8b\" (UniqueName: \"kubernetes.io/projected/5136dce1-1b36-4376-86f0-2d3f8638b07b-kube-api-access-j5d8b\") pod \"limitador-limitador-8588749977-7ddsw\" (UID: \"5136dce1-1b36-4376-86f0-2d3f8638b07b\") " pod="kuadrant-system/limitador-limitador-8588749977-7ddsw"
Sep 17 23:14:07.796468 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.796448    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5136dce1-1b36-4376-86f0-2d3f8638b07b-config-file\") pod \"limitador-limitador-8588749977-7ddsw\" (UID: \"5136dce1-1b36-4376-86f0-2d3f8638b07b\") " pod="kuadrant-system/limitador-limitador-8588749977-7ddsw"
Sep 17 23:14:07.803896 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.803856    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5d8b\" (UniqueName: \"kubernetes.io/projected/5136dce1-1b36-4376-86f0-2d3f8638b07b-kube-api-access-j5d8b\") pod \"limitador-limitador-8588749977-7ddsw\" (UID: \"5136dce1-1b36-4376-86f0-2d3f8638b07b\") " pod="kuadrant-system/limitador-limitador-8588749977-7ddsw"
Sep 17 23:14:07.822200 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.822171    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:07.947331 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:07.947234    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7ddsw"
Sep 17 23:14:08.217343 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:08.217305    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7ddsw"]
Sep 17 23:14:08.220161 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:14:08.220135    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5136dce1_1b36_4376_86f0_2d3f8638b07b.slice/crio-afb62bd120cabb51fc40e8ee0db167b273c3d1bbf1be2a143cf93d7a43aee55d WatchSource:0}: Error finding container afb62bd120cabb51fc40e8ee0db167b273c3d1bbf1be2a143cf93d7a43aee55d: Status 404 returned error can't find the container with id afb62bd120cabb51fc40e8ee0db167b273c3d1bbf1be2a143cf93d7a43aee55d
Sep 17 23:14:08.447130 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:08.447096    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mqd2t"]
Sep 17 23:14:08.449926 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:08.449908    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mqd2t"
Sep 17 23:14:08.452140 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:08.452116    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vbwcr\""
Sep 17 23:14:08.458249 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:08.458225    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mqd2t"]
Sep 17 23:14:08.479684 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:08.479605    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:08.504604 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:08.504569    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npzvz\" (UniqueName: \"kubernetes.io/projected/2660fe8a-cc19-4b51-bbc4-5b25cc092118-kube-api-access-npzvz\") pod \"authorino-7bcfd8f9cd-mqd2t\" (UID: \"2660fe8a-cc19-4b51-bbc4-5b25cc092118\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mqd2t"
Sep 17 23:14:08.560086 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:08.560053    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:08.605424 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:08.605383    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npzvz\" (UniqueName: \"kubernetes.io/projected/2660fe8a-cc19-4b51-bbc4-5b25cc092118-kube-api-access-npzvz\") pod \"authorino-7bcfd8f9cd-mqd2t\" (UID: \"2660fe8a-cc19-4b51-bbc4-5b25cc092118\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mqd2t"
Sep 17 23:14:08.612766 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:08.612742    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npzvz\" (UniqueName: \"kubernetes.io/projected/2660fe8a-cc19-4b51-bbc4-5b25cc092118-kube-api-access-npzvz\") pod \"authorino-7bcfd8f9cd-mqd2t\" (UID: \"2660fe8a-cc19-4b51-bbc4-5b25cc092118\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mqd2t"
Sep 17 23:14:08.760358 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:08.760266    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mqd2t"
Sep 17 23:14:09.005160 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:09.005122    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7ddsw" event={"ID":"5136dce1-1b36-4376-86f0-2d3f8638b07b","Type":"ContainerStarted","Data":"afb62bd120cabb51fc40e8ee0db167b273c3d1bbf1be2a143cf93d7a43aee55d"}
Sep 17 23:14:09.026978 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:09.026906    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mqd2t"]
Sep 17 23:14:09.029116 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:14:09.029090    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2660fe8a_cc19_4b51_bbc4_5b25cc092118.slice/crio-f4334368ab4be8dfd782631e1618d24be7cfcf9c143bc4046198d3b2275ad0d9 WatchSource:0}: Error finding container f4334368ab4be8dfd782631e1618d24be7cfcf9c143bc4046198d3b2275ad0d9: Status 404 returned error can't find the container with id f4334368ab4be8dfd782631e1618d24be7cfcf9c143bc4046198d3b2275ad0d9
Sep 17 23:14:10.012414 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:10.012357    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7ddsw" event={"ID":"5136dce1-1b36-4376-86f0-2d3f8638b07b","Type":"ContainerStarted","Data":"3b370e330e5f7f4494e7d84de0e8f63331f335d6d0e20c1d41415085b98fe2fc"}
Sep 17 23:14:10.012904 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:10.012429    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-7ddsw"
Sep 17 23:14:10.013818 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:10.013785    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mqd2t" event={"ID":"2660fe8a-cc19-4b51-bbc4-5b25cc092118","Type":"ContainerStarted","Data":"f4334368ab4be8dfd782631e1618d24be7cfcf9c143bc4046198d3b2275ad0d9"}
Sep 17 23:14:10.030705 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:10.030640    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-7ddsw" podStartSLOduration=1.800702105 podStartE2EDuration="3.030619511s" totalImagesPullingTime="1.229917406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:10.028906202 +0000 UTC m=+448.252581950" watchObservedRunningTime="2026-09-17 23:14:10.030619511 +0000 UTC m=+448.254295258"
Sep 17 23:14:10.182009 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:10.181974    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:10.284244 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:10.284207    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:11.020290 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:11.020247    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mqd2t" event={"ID":"2660fe8a-cc19-4b51-bbc4-5b25cc092118","Type":"ContainerStarted","Data":"82c5515fc0ce3d43ce90d22d523c6ec6a9817310156bda2e15572f3a2a2009f8"}
Sep 17 23:14:11.035764 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:11.035713    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-mqd2t" podStartSLOduration=1.687409809 podStartE2EDuration="3.035696559s" totalImagesPullingTime="1.34828675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:11.033603339 +0000 UTC m=+449.257279098" watchObservedRunningTime="2026-09-17 23:14:11.035696559 +0000 UTC m=+449.259372307"
Sep 17 23:14:11.704599 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:11.704564    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mqd2t"]
Sep 17 23:14:13.030264 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:13.030227    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-mqd2t" podUID="2660fe8a-cc19-4b51-bbc4-5b25cc092118" containerName="authorino" containerID="cri-o://82c5515fc0ce3d43ce90d22d523c6ec6a9817310156bda2e15572f3a2a2009f8" gracePeriod=30
Sep 17 23:14:13.275956 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:13.275930    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mqd2t"
Sep 17 23:14:13.348316 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:13.348222    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npzvz\" (UniqueName: \"kubernetes.io/projected/2660fe8a-cc19-4b51-bbc4-5b25cc092118-kube-api-access-npzvz\") pod \"2660fe8a-cc19-4b51-bbc4-5b25cc092118\" (UID: \"2660fe8a-cc19-4b51-bbc4-5b25cc092118\") "
Sep 17 23:14:13.350418 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:13.350386    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2660fe8a-cc19-4b51-bbc4-5b25cc092118-kube-api-access-npzvz" (OuterVolumeSpecName: "kube-api-access-npzvz") pod "2660fe8a-cc19-4b51-bbc4-5b25cc092118" (UID: "2660fe8a-cc19-4b51-bbc4-5b25cc092118"). InnerVolumeSpecName "kube-api-access-npzvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:14:13.448997 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:13.448961    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npzvz\" (UniqueName: \"kubernetes.io/projected/2660fe8a-cc19-4b51-bbc4-5b25cc092118-kube-api-access-npzvz\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:14:14.035407 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:14.035373    2568 generic.go:356] "Generic (PLEG): container finished" podID="2660fe8a-cc19-4b51-bbc4-5b25cc092118" containerID="82c5515fc0ce3d43ce90d22d523c6ec6a9817310156bda2e15572f3a2a2009f8" exitCode=0
Sep 17 23:14:14.035900 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:14.035422    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mqd2t" event={"ID":"2660fe8a-cc19-4b51-bbc4-5b25cc092118","Type":"ContainerDied","Data":"82c5515fc0ce3d43ce90d22d523c6ec6a9817310156bda2e15572f3a2a2009f8"}
Sep 17 23:14:14.035900 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:14.035457    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mqd2t" event={"ID":"2660fe8a-cc19-4b51-bbc4-5b25cc092118","Type":"ContainerDied","Data":"f4334368ab4be8dfd782631e1618d24be7cfcf9c143bc4046198d3b2275ad0d9"}
Sep 17 23:14:14.035900 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:14.035475    2568 scope.go:117] "RemoveContainer" containerID="82c5515fc0ce3d43ce90d22d523c6ec6a9817310156bda2e15572f3a2a2009f8"
Sep 17 23:14:14.035900 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:14.035480    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mqd2t"
Sep 17 23:14:14.044592 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:14.044574    2568 scope.go:117] "RemoveContainer" containerID="82c5515fc0ce3d43ce90d22d523c6ec6a9817310156bda2e15572f3a2a2009f8"
Sep 17 23:14:14.044828 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:14:14.044807    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c5515fc0ce3d43ce90d22d523c6ec6a9817310156bda2e15572f3a2a2009f8\": container with ID starting with 82c5515fc0ce3d43ce90d22d523c6ec6a9817310156bda2e15572f3a2a2009f8 not found: ID does not exist" containerID="82c5515fc0ce3d43ce90d22d523c6ec6a9817310156bda2e15572f3a2a2009f8"
Sep 17 23:14:14.044920 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:14.044841    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c5515fc0ce3d43ce90d22d523c6ec6a9817310156bda2e15572f3a2a2009f8"} err="failed to get container status \"82c5515fc0ce3d43ce90d22d523c6ec6a9817310156bda2e15572f3a2a2009f8\": rpc error: code = NotFound desc = could not find container \"82c5515fc0ce3d43ce90d22d523c6ec6a9817310156bda2e15572f3a2a2009f8\": container with ID starting with 82c5515fc0ce3d43ce90d22d523c6ec6a9817310156bda2e15572f3a2a2009f8 not found: ID does not exist"
Sep 17 23:14:14.050129 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:14.050104    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mqd2t"]
Sep 17 23:14:14.052419 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:14.052395    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mqd2t"]
Sep 17 23:14:15.989172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:15.989137    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2660fe8a-cc19-4b51-bbc4-5b25cc092118" path="/var/lib/kubelet/pods/2660fe8a-cc19-4b51-bbc4-5b25cc092118/volumes"
Sep 17 23:14:20.723658 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:20.723619    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7ddsw"]
Sep 17 23:14:20.724085 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:20.723820    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-7ddsw" podUID="5136dce1-1b36-4376-86f0-2d3f8638b07b" containerName="limitador" containerID="cri-o://3b370e330e5f7f4494e7d84de0e8f63331f335d6d0e20c1d41415085b98fe2fc" gracePeriod=30
Sep 17 23:14:20.724567 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:20.724547    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-7ddsw"
Sep 17 23:14:21.265527 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:21.265503    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7ddsw"
Sep 17 23:14:21.315610 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:21.315523    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5136dce1-1b36-4376-86f0-2d3f8638b07b-config-file\") pod \"5136dce1-1b36-4376-86f0-2d3f8638b07b\" (UID: \"5136dce1-1b36-4376-86f0-2d3f8638b07b\") "
Sep 17 23:14:21.315610 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:21.315586    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5d8b\" (UniqueName: \"kubernetes.io/projected/5136dce1-1b36-4376-86f0-2d3f8638b07b-kube-api-access-j5d8b\") pod \"5136dce1-1b36-4376-86f0-2d3f8638b07b\" (UID: \"5136dce1-1b36-4376-86f0-2d3f8638b07b\") "
Sep 17 23:14:21.315956 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:21.315937    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5136dce1-1b36-4376-86f0-2d3f8638b07b-config-file" (OuterVolumeSpecName: "config-file") pod "5136dce1-1b36-4376-86f0-2d3f8638b07b" (UID: "5136dce1-1b36-4376-86f0-2d3f8638b07b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:14:21.317718 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:21.317701    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5136dce1-1b36-4376-86f0-2d3f8638b07b-kube-api-access-j5d8b" (OuterVolumeSpecName: "kube-api-access-j5d8b") pod "5136dce1-1b36-4376-86f0-2d3f8638b07b" (UID: "5136dce1-1b36-4376-86f0-2d3f8638b07b"). InnerVolumeSpecName "kube-api-access-j5d8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:14:21.416537 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:21.416509    2568 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5136dce1-1b36-4376-86f0-2d3f8638b07b-config-file\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:14:21.416537 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:21.416534    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5d8b\" (UniqueName: \"kubernetes.io/projected/5136dce1-1b36-4376-86f0-2d3f8638b07b-kube-api-access-j5d8b\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:14:22.076558 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:22.076520    2568 generic.go:356] "Generic (PLEG): container finished" podID="5136dce1-1b36-4376-86f0-2d3f8638b07b" containerID="3b370e330e5f7f4494e7d84de0e8f63331f335d6d0e20c1d41415085b98fe2fc" exitCode=0
Sep 17 23:14:22.076558 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:22.076561    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7ddsw"
Sep 17 23:14:22.077072 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:22.076603    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7ddsw" event={"ID":"5136dce1-1b36-4376-86f0-2d3f8638b07b","Type":"ContainerDied","Data":"3b370e330e5f7f4494e7d84de0e8f63331f335d6d0e20c1d41415085b98fe2fc"}
Sep 17 23:14:22.077072 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:22.076660    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7ddsw" event={"ID":"5136dce1-1b36-4376-86f0-2d3f8638b07b","Type":"ContainerDied","Data":"afb62bd120cabb51fc40e8ee0db167b273c3d1bbf1be2a143cf93d7a43aee55d"}
Sep 17 23:14:22.077072 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:22.076687    2568 scope.go:117] "RemoveContainer" containerID="3b370e330e5f7f4494e7d84de0e8f63331f335d6d0e20c1d41415085b98fe2fc"
Sep 17 23:14:22.086200 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:22.086152    2568 scope.go:117] "RemoveContainer" containerID="3b370e330e5f7f4494e7d84de0e8f63331f335d6d0e20c1d41415085b98fe2fc"
Sep 17 23:14:22.086660 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:14:22.086632    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b370e330e5f7f4494e7d84de0e8f63331f335d6d0e20c1d41415085b98fe2fc\": container with ID starting with 3b370e330e5f7f4494e7d84de0e8f63331f335d6d0e20c1d41415085b98fe2fc not found: ID does not exist" containerID="3b370e330e5f7f4494e7d84de0e8f63331f335d6d0e20c1d41415085b98fe2fc"
Sep 17 23:14:22.086728 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:22.086668    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b370e330e5f7f4494e7d84de0e8f63331f335d6d0e20c1d41415085b98fe2fc"} err="failed to get container status \"3b370e330e5f7f4494e7d84de0e8f63331f335d6d0e20c1d41415085b98fe2fc\": rpc error: code = NotFound desc = could not find container \"3b370e330e5f7f4494e7d84de0e8f63331f335d6d0e20c1d41415085b98fe2fc\": container with ID starting with 3b370e330e5f7f4494e7d84de0e8f63331f335d6d0e20c1d41415085b98fe2fc not found: ID does not exist"
Sep 17 23:14:22.116394 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:22.116357    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7ddsw"]
Sep 17 23:14:22.126641 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:22.126612    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7ddsw"]
Sep 17 23:14:23.989965 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:23.989935    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5136dce1-1b36-4376-86f0-2d3f8638b07b" path="/var/lib/kubelet/pods/5136dce1-1b36-4376-86f0-2d3f8638b07b/volumes"
Sep 17 23:14:26.013742 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.013702    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:26.092762 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.092727    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:26.435752 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.435713    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-djm4s"]
Sep 17 23:14:26.436120 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.436102    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5136dce1-1b36-4376-86f0-2d3f8638b07b" containerName="limitador"
Sep 17 23:14:26.436206 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.436122    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5136dce1-1b36-4376-86f0-2d3f8638b07b" containerName="limitador"
Sep 17 23:14:26.436206 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.436153    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2660fe8a-cc19-4b51-bbc4-5b25cc092118" containerName="authorino"
Sep 17 23:14:26.436206 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.436162    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2660fe8a-cc19-4b51-bbc4-5b25cc092118" containerName="authorino"
Sep 17 23:14:26.436356 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.436249    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2660fe8a-cc19-4b51-bbc4-5b25cc092118" containerName="authorino"
Sep 17 23:14:26.436356 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.436264    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5136dce1-1b36-4376-86f0-2d3f8638b07b" containerName="limitador"
Sep 17 23:14:26.439343 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.439322    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-djm4s"
Sep 17 23:14:26.443548 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.443518    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 17 23:14:26.443848 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.443797    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 23:14:26.443848 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.443840    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-s9fqt\""
Sep 17 23:14:26.444664 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.444649    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 17 23:14:26.448064 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.448041    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-djm4s"]
Sep 17 23:14:26.469239 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.469209    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:26.564456 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.564424    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64bz5\" (UniqueName: \"kubernetes.io/projected/12b22b89-bf60-4d6b-bbeb-59d89590dde0-kube-api-access-64bz5\") pod \"postgres-69fb4594cf-djm4s\" (UID: \"12b22b89-bf60-4d6b-bbeb-59d89590dde0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-djm4s"
Sep 17 23:14:26.564582 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.564488    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12b22b89-bf60-4d6b-bbeb-59d89590dde0-data\") pod \"postgres-69fb4594cf-djm4s\" (UID: \"12b22b89-bf60-4d6b-bbeb-59d89590dde0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-djm4s"
Sep 17 23:14:26.665493 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.665453    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64bz5\" (UniqueName: \"kubernetes.io/projected/12b22b89-bf60-4d6b-bbeb-59d89590dde0-kube-api-access-64bz5\") pod \"postgres-69fb4594cf-djm4s\" (UID: \"12b22b89-bf60-4d6b-bbeb-59d89590dde0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-djm4s"
Sep 17 23:14:26.665656 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.665509    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12b22b89-bf60-4d6b-bbeb-59d89590dde0-data\") pod \"postgres-69fb4594cf-djm4s\" (UID: \"12b22b89-bf60-4d6b-bbeb-59d89590dde0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-djm4s"
Sep 17 23:14:26.665836 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.665821    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12b22b89-bf60-4d6b-bbeb-59d89590dde0-data\") pod \"postgres-69fb4594cf-djm4s\" (UID: \"12b22b89-bf60-4d6b-bbeb-59d89590dde0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-djm4s"
Sep 17 23:14:26.674400 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.674367    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64bz5\" (UniqueName: \"kubernetes.io/projected/12b22b89-bf60-4d6b-bbeb-59d89590dde0-kube-api-access-64bz5\") pod \"postgres-69fb4594cf-djm4s\" (UID: \"12b22b89-bf60-4d6b-bbeb-59d89590dde0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-djm4s"
Sep 17 23:14:26.751367 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:26.751271    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-djm4s"
Sep 17 23:14:27.035223 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:27.035195    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-djm4s"]
Sep 17 23:14:27.038173 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:14:27.038141    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b22b89_bf60_4d6b_bbeb_59d89590dde0.slice/crio-b531a354e4bd5d089387e8f721074dffa323584201226a677dca65a77efe3fa0 WatchSource:0}: Error finding container b531a354e4bd5d089387e8f721074dffa323584201226a677dca65a77efe3fa0: Status 404 returned error can't find the container with id b531a354e4bd5d089387e8f721074dffa323584201226a677dca65a77efe3fa0
Sep 17 23:14:27.101382 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:27.101345    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-djm4s" event={"ID":"12b22b89-bf60-4d6b-bbeb-59d89590dde0","Type":"ContainerStarted","Data":"b531a354e4bd5d089387e8f721074dffa323584201226a677dca65a77efe3fa0"}
Sep 17 23:14:32.131597 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:32.131565    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-djm4s" event={"ID":"12b22b89-bf60-4d6b-bbeb-59d89590dde0","Type":"ContainerStarted","Data":"05d251a896b4839d562318c39a9dbef1b4003e295c2caff6103297a6a552ef73"}
Sep 17 23:14:32.131979 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:32.131701    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-djm4s"
Sep 17 23:14:32.149152 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:32.149105    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-djm4s" podStartSLOduration=1.385158911 podStartE2EDuration="6.14908857s" totalImagesPullingTime="4.763929659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:32.147170684 +0000 UTC m=+470.370846434" watchObservedRunningTime="2026-09-17 23:14:32.14908857 +0000 UTC m=+470.372764318"
Sep 17 23:14:38.166324 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:38.166294    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-djm4s"
Sep 17 23:14:38.469773 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:38.469684    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:38.687449 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:38.687157    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-5dcqj"]
Sep 17 23:14:38.691621 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:38.691088    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5dcqj"
Sep 17 23:14:38.693937 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:38.693909    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vbwcr\""
Sep 17 23:14:38.697540 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:38.697514    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-5dcqj"]
Sep 17 23:14:38.879863 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:38.879827    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4vpl\" (UniqueName: \"kubernetes.io/projected/747ab4b3-b386-4e37-b42e-514342d5be5b-kube-api-access-q4vpl\") pod \"authorino-557f876f69-5dcqj\" (UID: \"747ab4b3-b386-4e37-b42e-514342d5be5b\") " pod="kuadrant-system/authorino-557f876f69-5dcqj"
Sep 17 23:14:38.909650 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:38.909612    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-5dcqj"]
Sep 17 23:14:38.909905 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:14:38.909884    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-q4vpl], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-5dcqj" podUID="747ab4b3-b386-4e37-b42e-514342d5be5b"
Sep 17 23:14:38.935512 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:38.935473    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-ff99c8464-cnmqr"]
Sep 17 23:14:38.937774 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:38.937757    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ff99c8464-cnmqr"
Sep 17 23:14:38.945789 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:38.945759    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-ff99c8464-cnmqr"]
Sep 17 23:14:38.980637 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:38.980601    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4vpl\" (UniqueName: \"kubernetes.io/projected/747ab4b3-b386-4e37-b42e-514342d5be5b-kube-api-access-q4vpl\") pod \"authorino-557f876f69-5dcqj\" (UID: \"747ab4b3-b386-4e37-b42e-514342d5be5b\") " pod="kuadrant-system/authorino-557f876f69-5dcqj"
Sep 17 23:14:38.988812 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:38.988780    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4vpl\" (UniqueName: \"kubernetes.io/projected/747ab4b3-b386-4e37-b42e-514342d5be5b-kube-api-access-q4vpl\") pod \"authorino-557f876f69-5dcqj\" (UID: \"747ab4b3-b386-4e37-b42e-514342d5be5b\") " pod="kuadrant-system/authorino-557f876f69-5dcqj"
Sep 17 23:14:39.081075 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:39.081031    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sdvs\" (UniqueName: \"kubernetes.io/projected/0e072ba5-899b-42c4-b73b-604b08293d02-kube-api-access-2sdvs\") pod \"authorino-ff99c8464-cnmqr\" (UID: \"0e072ba5-899b-42c4-b73b-604b08293d02\") " pod="kuadrant-system/authorino-ff99c8464-cnmqr"
Sep 17 23:14:39.163726 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:39.163634    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5dcqj"
Sep 17 23:14:39.169833 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:39.169809    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5dcqj"
Sep 17 23:14:39.182052 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:39.182025    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sdvs\" (UniqueName: \"kubernetes.io/projected/0e072ba5-899b-42c4-b73b-604b08293d02-kube-api-access-2sdvs\") pod \"authorino-ff99c8464-cnmqr\" (UID: \"0e072ba5-899b-42c4-b73b-604b08293d02\") " pod="kuadrant-system/authorino-ff99c8464-cnmqr"
Sep 17 23:14:39.189767 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:39.189742    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sdvs\" (UniqueName: \"kubernetes.io/projected/0e072ba5-899b-42c4-b73b-604b08293d02-kube-api-access-2sdvs\") pod \"authorino-ff99c8464-cnmqr\" (UID: \"0e072ba5-899b-42c4-b73b-604b08293d02\") " pod="kuadrant-system/authorino-ff99c8464-cnmqr"
Sep 17 23:14:39.247619 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:39.247584    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ff99c8464-cnmqr"
Sep 17 23:14:39.282663 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:39.282616    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4vpl\" (UniqueName: \"kubernetes.io/projected/747ab4b3-b386-4e37-b42e-514342d5be5b-kube-api-access-q4vpl\") pod \"747ab4b3-b386-4e37-b42e-514342d5be5b\" (UID: \"747ab4b3-b386-4e37-b42e-514342d5be5b\") "
Sep 17 23:14:39.285061 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:39.285027    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747ab4b3-b386-4e37-b42e-514342d5be5b-kube-api-access-q4vpl" (OuterVolumeSpecName: "kube-api-access-q4vpl") pod "747ab4b3-b386-4e37-b42e-514342d5be5b" (UID: "747ab4b3-b386-4e37-b42e-514342d5be5b"). InnerVolumeSpecName "kube-api-access-q4vpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:14:39.294182 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:39.294141    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:39.384663 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:39.384626    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4vpl\" (UniqueName: \"kubernetes.io/projected/747ab4b3-b386-4e37-b42e-514342d5be5b-kube-api-access-q4vpl\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:14:39.514138 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:39.514040    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-ff99c8464-cnmqr"]
Sep 17 23:14:39.516374 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:14:39.516351    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e072ba5_899b_42c4_b73b_604b08293d02.slice/crio-5653d14101bbda8869aecd63a271ec86da6f0f4a5c107f6f556c83e724710721 WatchSource:0}: Error finding container 5653d14101bbda8869aecd63a271ec86da6f0f4a5c107f6f556c83e724710721: Status 404 returned error can't find the container with id 5653d14101bbda8869aecd63a271ec86da6f0f4a5c107f6f556c83e724710721
Sep 17 23:14:40.170750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.170473    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5dcqj"
Sep 17 23:14:40.170750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.170510    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ff99c8464-cnmqr" event={"ID":"0e072ba5-899b-42c4-b73b-604b08293d02","Type":"ContainerStarted","Data":"1f8f5dda1f3ede16605102fcb04cf6f26d9271fdcaaa882e34b4e50b30f15cac"}
Sep 17 23:14:40.170750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.170550    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ff99c8464-cnmqr" event={"ID":"0e072ba5-899b-42c4-b73b-604b08293d02","Type":"ContainerStarted","Data":"5653d14101bbda8869aecd63a271ec86da6f0f4a5c107f6f556c83e724710721"}
Sep 17 23:14:40.188581 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.188523    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-ff99c8464-cnmqr" podStartSLOduration=1.637790971 podStartE2EDuration="2.188506419s" totalImagesPullingTime="550.715448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:40.18722813 +0000 UTC m=+478.410903879" watchObservedRunningTime="2026-09-17 23:14:40.188506419 +0000 UTC m=+478.412182167"
Sep 17 23:14:40.208604 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.208575    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-5dcqj"]
Sep 17 23:14:40.214826 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.214792    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-5dcqj"]
Sep 17 23:14:40.287083 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.287052    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f8f6d7d6b-rlqdj"]
Sep 17 23:14:40.289507 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.289481    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj"
Sep 17 23:14:40.291749 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.291728    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 23:14:40.297218 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.297195    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f8f6d7d6b-rlqdj"]
Sep 17 23:14:40.392513 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.392470    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qnt9\" (UniqueName: \"kubernetes.io/projected/feb07108-114e-4c2d-8479-3bf0df38087c-kube-api-access-7qnt9\") pod \"authorino-6f8f6d7d6b-rlqdj\" (UID: \"feb07108-114e-4c2d-8479-3bf0df38087c\") " pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj"
Sep 17 23:14:40.392711 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.392609    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/feb07108-114e-4c2d-8479-3bf0df38087c-tls-cert\") pod \"authorino-6f8f6d7d6b-rlqdj\" (UID: \"feb07108-114e-4c2d-8479-3bf0df38087c\") " pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj"
Sep 17 23:14:40.493078 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.493038    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/feb07108-114e-4c2d-8479-3bf0df38087c-tls-cert\") pod \"authorino-6f8f6d7d6b-rlqdj\" (UID: \"feb07108-114e-4c2d-8479-3bf0df38087c\") " pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj"
Sep 17 23:14:40.493255 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.493097    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qnt9\" (UniqueName: \"kubernetes.io/projected/feb07108-114e-4c2d-8479-3bf0df38087c-kube-api-access-7qnt9\") pod \"authorino-6f8f6d7d6b-rlqdj\" (UID: \"feb07108-114e-4c2d-8479-3bf0df38087c\") " pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj"
Sep 17 23:14:40.495821 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.495799    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/feb07108-114e-4c2d-8479-3bf0df38087c-tls-cert\") pod \"authorino-6f8f6d7d6b-rlqdj\" (UID: \"feb07108-114e-4c2d-8479-3bf0df38087c\") " pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj"
Sep 17 23:14:40.501987 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.501959    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qnt9\" (UniqueName: \"kubernetes.io/projected/feb07108-114e-4c2d-8479-3bf0df38087c-kube-api-access-7qnt9\") pod \"authorino-6f8f6d7d6b-rlqdj\" (UID: \"feb07108-114e-4c2d-8479-3bf0df38087c\") " pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj"
Sep 17 23:14:40.599395 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.599298    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj"
Sep 17 23:14:40.867288 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:40.867262    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f8f6d7d6b-rlqdj"]
Sep 17 23:14:40.869391 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:14:40.869367    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfeb07108_114e_4c2d_8479_3bf0df38087c.slice/crio-79ce7fe88336668a03444f757c30e00f34a009f2cbddb20a04dddbd52a6c2a4d WatchSource:0}: Error finding container 79ce7fe88336668a03444f757c30e00f34a009f2cbddb20a04dddbd52a6c2a4d: Status 404 returned error can't find the container with id 79ce7fe88336668a03444f757c30e00f34a009f2cbddb20a04dddbd52a6c2a4d
Sep 17 23:14:41.176586 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:41.176500    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj" event={"ID":"feb07108-114e-4c2d-8479-3bf0df38087c","Type":"ContainerStarted","Data":"79ce7fe88336668a03444f757c30e00f34a009f2cbddb20a04dddbd52a6c2a4d"}
Sep 17 23:14:41.990514 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:41.990431    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="747ab4b3-b386-4e37-b42e-514342d5be5b" path="/var/lib/kubelet/pods/747ab4b3-b386-4e37-b42e-514342d5be5b/volumes"
Sep 17 23:14:42.182624 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:42.182585    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj" event={"ID":"feb07108-114e-4c2d-8479-3bf0df38087c","Type":"ContainerStarted","Data":"0464e27416a28b903dd0cfa03e825a837c0ab571aa308a4916c3a3e5994e7bd9"}
Sep 17 23:14:42.199556 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:42.199495    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj" podStartSLOduration=1.476080659 podStartE2EDuration="2.199480291s" totalImagesPullingTime="723.399632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:42.198529805 +0000 UTC m=+480.422205554" watchObservedRunningTime="2026-09-17 23:14:42.199480291 +0000 UTC m=+480.423156039"
Sep 17 23:14:42.222745 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:42.222709    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-ff99c8464-cnmqr"]
Sep 17 23:14:42.222940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:42.222893    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-ff99c8464-cnmqr" podUID="0e072ba5-899b-42c4-b73b-604b08293d02" containerName="authorino" containerID="cri-o://1f8f5dda1f3ede16605102fcb04cf6f26d9271fdcaaa882e34b4e50b30f15cac" gracePeriod=30
Sep 17 23:14:42.502094 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:42.502060    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ff99c8464-cnmqr"
Sep 17 23:14:42.613186 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:42.613150    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sdvs\" (UniqueName: \"kubernetes.io/projected/0e072ba5-899b-42c4-b73b-604b08293d02-kube-api-access-2sdvs\") pod \"0e072ba5-899b-42c4-b73b-604b08293d02\" (UID: \"0e072ba5-899b-42c4-b73b-604b08293d02\") "
Sep 17 23:14:42.615242 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:42.615207    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e072ba5-899b-42c4-b73b-604b08293d02-kube-api-access-2sdvs" (OuterVolumeSpecName: "kube-api-access-2sdvs") pod "0e072ba5-899b-42c4-b73b-604b08293d02" (UID: "0e072ba5-899b-42c4-b73b-604b08293d02"). InnerVolumeSpecName "kube-api-access-2sdvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:14:42.714837 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:42.714795    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sdvs\" (UniqueName: \"kubernetes.io/projected/0e072ba5-899b-42c4-b73b-604b08293d02-kube-api-access-2sdvs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:14:43.190380 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:43.190341    2568 generic.go:356] "Generic (PLEG): container finished" podID="0e072ba5-899b-42c4-b73b-604b08293d02" containerID="1f8f5dda1f3ede16605102fcb04cf6f26d9271fdcaaa882e34b4e50b30f15cac" exitCode=0
Sep 17 23:14:43.190784 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:43.190421    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ff99c8464-cnmqr"
Sep 17 23:14:43.190784 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:43.190425    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ff99c8464-cnmqr" event={"ID":"0e072ba5-899b-42c4-b73b-604b08293d02","Type":"ContainerDied","Data":"1f8f5dda1f3ede16605102fcb04cf6f26d9271fdcaaa882e34b4e50b30f15cac"}
Sep 17 23:14:43.190784 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:43.190469    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-ff99c8464-cnmqr" event={"ID":"0e072ba5-899b-42c4-b73b-604b08293d02","Type":"ContainerDied","Data":"5653d14101bbda8869aecd63a271ec86da6f0f4a5c107f6f556c83e724710721"}
Sep 17 23:14:43.190784 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:43.190488    2568 scope.go:117] "RemoveContainer" containerID="1f8f5dda1f3ede16605102fcb04cf6f26d9271fdcaaa882e34b4e50b30f15cac"
Sep 17 23:14:43.202706 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:43.202688    2568 scope.go:117] "RemoveContainer" containerID="1f8f5dda1f3ede16605102fcb04cf6f26d9271fdcaaa882e34b4e50b30f15cac"
Sep 17 23:14:43.203102 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:14:43.203081    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8f5dda1f3ede16605102fcb04cf6f26d9271fdcaaa882e34b4e50b30f15cac\": container with ID starting with 1f8f5dda1f3ede16605102fcb04cf6f26d9271fdcaaa882e34b4e50b30f15cac not found: ID does not exist" containerID="1f8f5dda1f3ede16605102fcb04cf6f26d9271fdcaaa882e34b4e50b30f15cac"
Sep 17 23:14:43.203195 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:43.203114    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8f5dda1f3ede16605102fcb04cf6f26d9271fdcaaa882e34b4e50b30f15cac"} err="failed to get container status \"1f8f5dda1f3ede16605102fcb04cf6f26d9271fdcaaa882e34b4e50b30f15cac\": rpc error: code = NotFound desc = could not find container \"1f8f5dda1f3ede16605102fcb04cf6f26d9271fdcaaa882e34b4e50b30f15cac\": container with ID starting with 1f8f5dda1f3ede16605102fcb04cf6f26d9271fdcaaa882e34b4e50b30f15cac not found: ID does not exist"
Sep 17 23:14:43.214520 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:43.214490    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-ff99c8464-cnmqr"]
Sep 17 23:14:43.217217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:43.217190    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-ff99c8464-cnmqr"]
Sep 17 23:14:43.992400 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:43.992360    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e072ba5-899b-42c4-b73b-604b08293d02" path="/var/lib/kubelet/pods/0e072ba5-899b-42c4-b73b-604b08293d02/volumes"
Sep 17 23:14:44.853947 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:44.853913    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:44.925322 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:44.925293    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:44.980983 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:44.980918    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:45.072681 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:45.072644    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:45.524419 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:45.524388    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:48.923543 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:48.923498    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:49.023326 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:14:49.023292    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:19.134581 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:19.134535    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:19.229297 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:19.229267    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:20.630697 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.630666    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:20.729721 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.729686    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:20.827725 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.827691    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:20.846810 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.846773    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"]
Sep 17 23:15:20.847158 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.847145    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e072ba5-899b-42c4-b73b-604b08293d02" containerName="authorino"
Sep 17 23:15:20.847208 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.847160    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e072ba5-899b-42c4-b73b-604b08293d02" containerName="authorino"
Sep 17 23:15:20.847243 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.847212    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e072ba5-899b-42c4-b73b-604b08293d02" containerName="authorino"
Sep 17 23:15:20.850389 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.850372    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:20.853097 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.853076    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cqmcq\""
Sep 17 23:15:20.854028 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.854010    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 17 23:15:20.854134 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.854010    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 17 23:15:20.863419 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.863393    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"]
Sep 17 23:15:20.929245 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.929166    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:20.935257 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.935226    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-maas-api-tls\") pod \"maas-api-5d54fc9767-4jnj6\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:20.935393 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.935261    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78772\" (UniqueName: \"kubernetes.io/projected/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-kube-api-access-78772\") pod \"maas-api-5d54fc9767-4jnj6\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:20.935393 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.935289    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-metrics-certs\") pod \"maas-api-5d54fc9767-4jnj6\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:20.935480 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:20.935444    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-openshift-service-ca\") pod \"maas-api-5d54fc9767-4jnj6\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:21.036039 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:21.036001    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-openshift-service-ca\") pod \"maas-api-5d54fc9767-4jnj6\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:21.036234 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:21.036056    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-maas-api-tls\") pod \"maas-api-5d54fc9767-4jnj6\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:21.036234 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:21.036074    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78772\" (UniqueName: \"kubernetes.io/projected/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-kube-api-access-78772\") pod \"maas-api-5d54fc9767-4jnj6\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:21.036234 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:21.036091    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-metrics-certs\") pod \"maas-api-5d54fc9767-4jnj6\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:21.036730 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:21.036705    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-openshift-service-ca\") pod \"maas-api-5d54fc9767-4jnj6\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:21.038472 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:21.038453    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-maas-api-tls\") pod \"maas-api-5d54fc9767-4jnj6\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:21.038647 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:21.038609    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-metrics-certs\") pod \"maas-api-5d54fc9767-4jnj6\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:21.044619 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:21.044591    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78772\" (UniqueName: \"kubernetes.io/projected/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-kube-api-access-78772\") pod \"maas-api-5d54fc9767-4jnj6\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") " pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:21.161126 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:21.161095    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:21.278827 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:21.278796    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:21.375859 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:21.375822    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:21.436790 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:21.436758    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"]
Sep 17 23:15:21.438811 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:15:21.438780    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60afdc58_a3b9_4b21_ab79_ffd0ca31801c.slice/crio-c39268ecbdfa8a3f2b4efa255b67d6400393fc7190ee31d563b1aa9f063024cd WatchSource:0}: Error finding container c39268ecbdfa8a3f2b4efa255b67d6400393fc7190ee31d563b1aa9f063024cd: Status 404 returned error can't find the container with id c39268ecbdfa8a3f2b4efa255b67d6400393fc7190ee31d563b1aa9f063024cd
Sep 17 23:15:22.367530 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:22.367460    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6" event={"ID":"60afdc58-a3b9-4b21-ab79-ffd0ca31801c","Type":"ContainerStarted","Data":"c39268ecbdfa8a3f2b4efa255b67d6400393fc7190ee31d563b1aa9f063024cd"}
Sep 17 23:15:24.379122 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:24.379080    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6" event={"ID":"60afdc58-a3b9-4b21-ab79-ffd0ca31801c","Type":"ContainerStarted","Data":"9d64d97112ec522e832620290ce03d339d6e98c899d3c30d481751cfc4676ed6"}
Sep 17 23:15:24.379617 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:24.379170    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:24.398843 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:24.398792    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6" podStartSLOduration=1.975597406 podStartE2EDuration="4.398775336s" totalImagesPullingTime="2.42317793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:24.397128339 +0000 UTC m=+522.620804098" watchObservedRunningTime="2026-09-17 23:15:24.398775336 +0000 UTC m=+522.622451084"
Sep 17 23:15:30.389253 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:30.389225    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:15:31.619691 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:31.619651    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:31.719802 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:31.719765    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:52.620911 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:15:52.620839    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:03.294113 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.294077    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"]
Sep 17 23:16:03.298615 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.298591    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.301743 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.301724    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 23:16:03.301927 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.301910    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 17 23:16:03.302778 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.302754    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 23:16:03.302891 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.302861    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wkppn\""
Sep 17 23:16:03.315740 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.315715    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"]
Sep 17 23:16:03.361326 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.361280    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:03.394328 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.394298    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.394328 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.394329    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.394522 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.394372    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vchc2\" (UniqueName: \"kubernetes.io/projected/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-kube-api-access-vchc2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.394522 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.394393    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.394522 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.394428    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.394522 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.394459    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.495260 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.495218    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.495452 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.495279    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vchc2\" (UniqueName: \"kubernetes.io/projected/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-kube-api-access-vchc2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.495452 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.495302    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.495452 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.495326    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.495452 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.495362    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.495452 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.495393    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.495733 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.495695    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.495803 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.495780    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.495851 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.495813    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.497614 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.497592    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.497778 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.497758    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.502424 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.502407    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vchc2\" (UniqueName: \"kubernetes.io/projected/0a4e0339-0272-4cfd-92f2-e1aaad8d867b-kube-api-access-vchc2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rqzg\" (UID: \"0a4e0339-0272-4cfd-92f2-e1aaad8d867b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.608846 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.608739    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:03.877840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:03.877813    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"]
Sep 17 23:16:03.879763 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:16:03.879732    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4e0339_0272_4cfd_92f2_e1aaad8d867b.slice/crio-e6ce99efdeb4ee7e059f4e7cbcf27798adb1ddf3330b84e713e5a99da787992f WatchSource:0}: Error finding container e6ce99efdeb4ee7e059f4e7cbcf27798adb1ddf3330b84e713e5a99da787992f: Status 404 returned error can't find the container with id e6ce99efdeb4ee7e059f4e7cbcf27798adb1ddf3330b84e713e5a99da787992f
Sep 17 23:16:04.566835 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:04.566790    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg" event={"ID":"0a4e0339-0272-4cfd-92f2-e1aaad8d867b","Type":"ContainerStarted","Data":"e6ce99efdeb4ee7e059f4e7cbcf27798adb1ddf3330b84e713e5a99da787992f"}
Sep 17 23:16:05.573030 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:05.572995    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg" event={"ID":"0a4e0339-0272-4cfd-92f2-e1aaad8d867b","Type":"ContainerStarted","Data":"04eb217086df5beffdbff708ee59287817ceed220802b6eafefe92c5fd8f3479"}
Sep 17 23:16:05.590949 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:05.590894    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg" podStartSLOduration=1.531946164 podStartE2EDuration="2.590860162s" totalImagesPullingTime="1.058913998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:05.58963864 +0000 UTC m=+563.813314390" watchObservedRunningTime="2026-09-17 23:16:05.590860162 +0000 UTC m=+563.814535909"
Sep 17 23:16:06.937480 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:06.937448    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:06.987886 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:06.987833    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:09.998600 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:09.998571    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:13.609439 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:13.609412    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:13.609818 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:13.609490    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:13.622185 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:13.622152    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:14.624953 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:14.624920    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rqzg"
Sep 17 23:16:27.408027 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.407993    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"]
Sep 17 23:16:27.410566 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.410546    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.413009 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.412986    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 17 23:16:27.421283 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.421257    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"]
Sep 17 23:16:27.438838 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.438806    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:27.518117 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.518074    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.518117 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.518116    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.518365 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.518146    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.518365 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.518191    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.518365 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.518221    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tmdt\" (UniqueName: \"kubernetes.io/projected/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-kube-api-access-7tmdt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.518365 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.518264    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.619441 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.619409    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tmdt\" (UniqueName: \"kubernetes.io/projected/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-kube-api-access-7tmdt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.619634 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.619452    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.619634 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.619517    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.619634 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.619536    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.619634 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.619575    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.619634 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.619607    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.620011 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.619989    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.620079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.620036    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.620116 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.620081    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.621817 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.621800    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.622166 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.622149    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.627245 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.627215    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tmdt\" (UniqueName: \"kubernetes.io/projected/bb3255ee-5e56-46ef-a02d-4ba9dce85c2b-kube-api-access-7tmdt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-q5g6s\" (UID: \"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.721004 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.720915    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:27.990830 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:27.990795    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"]
Sep 17 23:16:27.991796 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:16:27.991771    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3255ee_5e56_46ef_a02d_4ba9dce85c2b.slice/crio-4841aed19afb875396353707c3ef476f51e1a033e156ab47f85b1f39373017ea WatchSource:0}: Error finding container 4841aed19afb875396353707c3ef476f51e1a033e156ab47f85b1f39373017ea: Status 404 returned error can't find the container with id 4841aed19afb875396353707c3ef476f51e1a033e156ab47f85b1f39373017ea
Sep 17 23:16:28.676264 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:28.676222    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s" event={"ID":"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b","Type":"ContainerStarted","Data":"37046744dfdfbb47a47ae2ecb3c04642feff0325a483cc1b24fb0371f8cdd0ef"}
Sep 17 23:16:28.676647 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:28.676281    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s" event={"ID":"bb3255ee-5e56-46ef-a02d-4ba9dce85c2b","Type":"ContainerStarted","Data":"4841aed19afb875396353707c3ef476f51e1a033e156ab47f85b1f39373017ea"}
Sep 17 23:16:28.693657 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:28.693554    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s" podStartSLOduration=1.512888075 podStartE2EDuration="1.693532287s" totalImagesPullingTime="180.644212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:28.693399837 +0000 UTC m=+586.917075588" watchObservedRunningTime="2026-09-17 23:16:28.693532287 +0000 UTC m=+586.917208049"
Sep 17 23:16:32.890304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:32.890267    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"]
Sep 17 23:16:32.892833 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:32.892812    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:32.895356 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:32.895332    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 17 23:16:32.904746 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:32.904720    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"]
Sep 17 23:16:32.920789 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:32.920762    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:32.960545 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:32.960514    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:32.960697 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:32.960571    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:32.960697 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:32.960589    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:32.960813 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:32.960734    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:32.960813 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:32.960773    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmwq\" (UniqueName: \"kubernetes.io/projected/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-kube-api-access-nqmwq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:32.960943 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:32.960834    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:33.062161 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.062125    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:33.062161 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.062162    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmwq\" (UniqueName: \"kubernetes.io/projected/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-kube-api-access-nqmwq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:33.062434 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.062211    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:33.062434 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.062368    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:33.062547 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.062452    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:33.062547 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.062479    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:33.062708 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.062682    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:33.062776 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.062725    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:33.062832 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.062812    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:33.064490 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.064467    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:33.064688 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.064670    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:33.069799 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.069774    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmwq\" (UniqueName: \"kubernetes.io/projected/a208e26b-f4d8-4a8c-9dc8-961b0b2c6781-kube-api-access-nqmwq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl\" (UID: \"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:33.204376 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.204282    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:33.473184 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.473158    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"]
Sep 17 23:16:33.475362 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:16:33.475336    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda208e26b_f4d8_4a8c_9dc8_961b0b2c6781.slice/crio-184644de09fd878b7c18342238aecffab460dee3f8248b5797e0b68ad745cf85 WatchSource:0}: Error finding container 184644de09fd878b7c18342238aecffab460dee3f8248b5797e0b68ad745cf85: Status 404 returned error can't find the container with id 184644de09fd878b7c18342238aecffab460dee3f8248b5797e0b68ad745cf85
Sep 17 23:16:33.700931 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.700900    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl" event={"ID":"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781","Type":"ContainerStarted","Data":"bd57a837cc441ffc49b7865d4b747dedb73578d65b733b38fb6fc71620e63bfd"}
Sep 17 23:16:33.701051 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.700955    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl" event={"ID":"a208e26b-f4d8-4a8c-9dc8-961b0b2c6781","Type":"ContainerStarted","Data":"184644de09fd878b7c18342238aecffab460dee3f8248b5797e0b68ad745cf85"}
Sep 17 23:16:33.718928 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:33.718879    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl" podStartSLOduration=1.559278247 podStartE2EDuration="1.718850434s" totalImagesPullingTime="159.572187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:33.718203435 +0000 UTC m=+591.941879184" watchObservedRunningTime="2026-09-17 23:16:33.718850434 +0000 UTC m=+591.942526181"
Sep 17 23:16:37.721853 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:37.721797    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:37.722438 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:37.721927    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:37.734814 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:37.734782    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:38.742428 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:38.742398    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-q5g6s"
Sep 17 23:16:38.889712 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:38.889675    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:38.991079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:38.991029    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:43.205375 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:43.205334    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:43.205768 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:43.205482    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:43.218334 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:43.218304    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:43.766096 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:43.766065    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl"
Sep 17 23:16:47.309909 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.309858    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"]
Sep 17 23:16:47.316387 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.316363    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.318653 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.318627    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 17 23:16:47.322406 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.322378    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"]
Sep 17 23:16:47.333126 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.333088    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:47.391622 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.391579    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1261b14-190b-4e6c-829a-1eab3a3142c0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.391622 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.391621    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8794\" (UniqueName: \"kubernetes.io/projected/e1261b14-190b-4e6c-829a-1eab3a3142c0-kube-api-access-q8794\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.391912 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.391662    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1261b14-190b-4e6c-829a-1eab3a3142c0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.391912 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.391699    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1261b14-190b-4e6c-829a-1eab3a3142c0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.391912 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.391714    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1261b14-190b-4e6c-829a-1eab3a3142c0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.391912 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.391730    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1261b14-190b-4e6c-829a-1eab3a3142c0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.492584 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.492548    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1261b14-190b-4e6c-829a-1eab3a3142c0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.492584 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.492591    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8794\" (UniqueName: \"kubernetes.io/projected/e1261b14-190b-4e6c-829a-1eab3a3142c0-kube-api-access-q8794\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.492836 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.492625    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1261b14-190b-4e6c-829a-1eab3a3142c0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.492836 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.492684    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1261b14-190b-4e6c-829a-1eab3a3142c0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.492836 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.492710    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1261b14-190b-4e6c-829a-1eab3a3142c0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.492836 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.492732    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1261b14-190b-4e6c-829a-1eab3a3142c0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.493076 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.492951    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1261b14-190b-4e6c-829a-1eab3a3142c0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.493134 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.493066    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1261b14-190b-4e6c-829a-1eab3a3142c0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.493134 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.493097    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1261b14-190b-4e6c-829a-1eab3a3142c0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.495029 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.495011    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1261b14-190b-4e6c-829a-1eab3a3142c0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.495403 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.495383    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1261b14-190b-4e6c-829a-1eab3a3142c0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.500078 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.500060    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8794\" (UniqueName: \"kubernetes.io/projected/e1261b14-190b-4e6c-829a-1eab3a3142c0-kube-api-access-q8794\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l6tbq\" (UID: \"e1261b14-190b-4e6c-829a-1eab3a3142c0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.629716 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.629616    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:47.900062 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:47.900009    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"]
Sep 17 23:16:47.902348 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:16:47.902319    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1261b14_190b_4e6c_829a_1eab3a3142c0.slice/crio-074d2086f140c16838d4ffca461b4f9f96a38e4b73f13a19546ff0f3a1a2a4ad WatchSource:0}: Error finding container 074d2086f140c16838d4ffca461b4f9f96a38e4b73f13a19546ff0f3a1a2a4ad: Status 404 returned error can't find the container with id 074d2086f140c16838d4ffca461b4f9f96a38e4b73f13a19546ff0f3a1a2a4ad
Sep 17 23:16:48.779650 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:48.779614    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq" event={"ID":"e1261b14-190b-4e6c-829a-1eab3a3142c0","Type":"ContainerStarted","Data":"9ad968a6b477db92407adbc11e0dfcc2f180a11def17783e16d625e68f653ea0"}
Sep 17 23:16:48.780143 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:48.779672    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq" event={"ID":"e1261b14-190b-4e6c-829a-1eab3a3142c0","Type":"ContainerStarted","Data":"074d2086f140c16838d4ffca461b4f9f96a38e4b73f13a19546ff0f3a1a2a4ad"}
Sep 17 23:16:48.798199 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:48.798086    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq" podStartSLOduration=1.6284780730000001 podStartE2EDuration="1.798068505s" totalImagesPullingTime="169.590432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:48.79627697 +0000 UTC m=+607.019952718" watchObservedRunningTime="2026-09-17 23:16:48.798068505 +0000 UTC m=+607.021744299"
Sep 17 23:16:57.629784 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:57.629694    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:57.629784 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:57.629760    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:57.642289 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:57.642261    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:57.839616 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:57.839582    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l6tbq"
Sep 17 23:16:58.494574 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.494538    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"]
Sep 17 23:16:58.498089 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.498064    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.500267 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.500237    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 17 23:16:58.506957 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.506930    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"]
Sep 17 23:16:58.522778 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.522741    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:58.612482 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.612444    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.612671 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.612520    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.612671 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.612566    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.612671 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.612590    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.612671 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.612635    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9ztm\" (UniqueName: \"kubernetes.io/projected/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-kube-api-access-w9ztm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.612808 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.612690    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.714185 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.714145    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.714575 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.714198    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.714575 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.714316    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.714575 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.714355    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9ztm\" (UniqueName: \"kubernetes.io/projected/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-kube-api-access-w9ztm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.714575 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.714438    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.714575 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.714480    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.714575 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.714556    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.714979 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.714672    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.714979 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.714838    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.716548 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.716527    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.716740 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.716720    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.721756 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.721738    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9ztm\" (UniqueName: \"kubernetes.io/projected/1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0-kube-api-access-w9ztm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tw79f\" (UID: \"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:58.808763 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:58.808722    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:16:59.079021 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:59.078995    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"]
Sep 17 23:16:59.080907 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:16:59.080851    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f9a8fbf_4cc9_4ed0_bd9f_730d8cb9f0e0.slice/crio-134d0b4a63b8807ce82d4ecba4c81dfe7f903b598847971f4fe42200eeb4309f WatchSource:0}: Error finding container 134d0b4a63b8807ce82d4ecba4c81dfe7f903b598847971f4fe42200eeb4309f: Status 404 returned error can't find the container with id 134d0b4a63b8807ce82d4ecba4c81dfe7f903b598847971f4fe42200eeb4309f
Sep 17 23:16:59.839711 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:59.839677    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f" event={"ID":"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0","Type":"ContainerStarted","Data":"5ca3a0cc8a898b82f86560f6b56e478a0321842345c66354973fd353d951aaa0"}
Sep 17 23:16:59.840121 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:59.839719    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f" event={"ID":"1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0","Type":"ContainerStarted","Data":"134d0b4a63b8807ce82d4ecba4c81dfe7f903b598847971f4fe42200eeb4309f"}
Sep 17 23:16:59.857675 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:16:59.857624    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f" podStartSLOduration=1.6866192230000001 podStartE2EDuration="1.857606567s" totalImagesPullingTime="170.987344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:59.856908272 +0000 UTC m=+618.080584014" watchObservedRunningTime="2026-09-17 23:16:59.857606567 +0000 UTC m=+618.081282315"
Sep 17 23:17:08.809859 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:17:08.809812    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:17:08.810431 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:17:08.809907    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:17:08.822416 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:17:08.822386    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:17:08.892454 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:17:08.892426    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tw79f"
Sep 17 23:17:27.295744 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:17:27.295672    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:16.997566 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:16.997500    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f8f6d7d6b-rlqdj"]
Sep 17 23:18:16.998278 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:16.997842    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj" podUID="feb07108-114e-4c2d-8479-3bf0df38087c" containerName="authorino" containerID="cri-o://0464e27416a28b903dd0cfa03e825a837c0ab571aa308a4916c3a3e5994e7bd9" gracePeriod=30
Sep 17 23:18:17.200921 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:17.200889    2568 generic.go:356] "Generic (PLEG): container finished" podID="feb07108-114e-4c2d-8479-3bf0df38087c" containerID="0464e27416a28b903dd0cfa03e825a837c0ab571aa308a4916c3a3e5994e7bd9" exitCode=0
Sep 17 23:18:17.201080 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:17.200946    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj" event={"ID":"feb07108-114e-4c2d-8479-3bf0df38087c","Type":"ContainerDied","Data":"0464e27416a28b903dd0cfa03e825a837c0ab571aa308a4916c3a3e5994e7bd9"}
Sep 17 23:18:17.245739 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:17.245715    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj"
Sep 17 23:18:17.369857 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:17.369819    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qnt9\" (UniqueName: \"kubernetes.io/projected/feb07108-114e-4c2d-8479-3bf0df38087c-kube-api-access-7qnt9\") pod \"feb07108-114e-4c2d-8479-3bf0df38087c\" (UID: \"feb07108-114e-4c2d-8479-3bf0df38087c\") "
Sep 17 23:18:17.370054 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:17.369903    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/feb07108-114e-4c2d-8479-3bf0df38087c-tls-cert\") pod \"feb07108-114e-4c2d-8479-3bf0df38087c\" (UID: \"feb07108-114e-4c2d-8479-3bf0df38087c\") "
Sep 17 23:18:17.371973 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:17.371946    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb07108-114e-4c2d-8479-3bf0df38087c-kube-api-access-7qnt9" (OuterVolumeSpecName: "kube-api-access-7qnt9") pod "feb07108-114e-4c2d-8479-3bf0df38087c" (UID: "feb07108-114e-4c2d-8479-3bf0df38087c"). InnerVolumeSpecName "kube-api-access-7qnt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:18:17.382552 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:17.382509    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feb07108-114e-4c2d-8479-3bf0df38087c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "feb07108-114e-4c2d-8479-3bf0df38087c" (UID: "feb07108-114e-4c2d-8479-3bf0df38087c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:18:17.471396 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:17.471358    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qnt9\" (UniqueName: \"kubernetes.io/projected/feb07108-114e-4c2d-8479-3bf0df38087c-kube-api-access-7qnt9\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:18:17.471396 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:17.471388    2568 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/feb07108-114e-4c2d-8479-3bf0df38087c-tls-cert\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:18:17.499321 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:17.499273    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:17.595644 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:17.595612    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:18.207406 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:18.207380    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj"
Sep 17 23:18:18.207809 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:18.207379    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f8f6d7d6b-rlqdj" event={"ID":"feb07108-114e-4c2d-8479-3bf0df38087c","Type":"ContainerDied","Data":"79ce7fe88336668a03444f757c30e00f34a009f2cbddb20a04dddbd52a6c2a4d"}
Sep 17 23:18:18.207809 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:18.207508    2568 scope.go:117] "RemoveContainer" containerID="0464e27416a28b903dd0cfa03e825a837c0ab571aa308a4916c3a3e5994e7bd9"
Sep 17 23:18:18.225675 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:18.225649    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f8f6d7d6b-rlqdj"]
Sep 17 23:18:18.238972 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:18.238946    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f8f6d7d6b-rlqdj"]
Sep 17 23:18:18.626295 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:18.626263    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:18.732834 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:18.732796    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:19.388152 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:19.388121    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:19.488568 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:19.488536    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:19.989522 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:19.989491    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feb07108-114e-4c2d-8479-3bf0df38087c" path="/var/lib/kubelet/pods/feb07108-114e-4c2d-8479-3bf0df38087c/volumes"
Sep 17 23:18:25.129876 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:25.129815    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:33.956878 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:33.956832    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:34.057099 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:34.057063    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:34.161214 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:34.161178    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:34.315032 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:34.315000    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:34.427139 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:34.427109    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:34.513138 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:34.513102    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:34.764938 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:34.764834    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:34.865386 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:34.865351    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:35.390812 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:35.390784    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:35.489092 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:35.489054    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:35.831485 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:35.831459    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:37.139539 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:37.139492    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:37.233293 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:37.233252    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:37.334487 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:37.334435    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:38.036555 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:38.036522    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:38.140637 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:38.140596    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:38.245043 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:38.245007    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:38.917338 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:38.917296    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"]
Sep 17 23:18:38.917772 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:38.917754    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb07108-114e-4c2d-8479-3bf0df38087c" containerName="authorino"
Sep 17 23:18:38.917881 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:38.917775    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb07108-114e-4c2d-8479-3bf0df38087c" containerName="authorino"
Sep 17 23:18:38.917945 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:38.917908    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="feb07108-114e-4c2d-8479-3bf0df38087c" containerName="authorino"
Sep 17 23:18:38.920966 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:38.920941    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:38.923441 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:38.923421    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-825f30-serving-cert\""
Sep 17 23:18:38.930632 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:38.930603    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"]
Sep 17 23:18:38.956278 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:38.956250    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:38.971708 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:38.971679    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f11e1b-28fb-4b8b-b6af-198c5decffd4-metrics-certs\") pod \"maas-api-e2e-worker-w4-825f30-5448495788-zbxs6\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:38.971908 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:38.971739    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fntnn\" (UniqueName: \"kubernetes.io/projected/71f11e1b-28fb-4b8b-b6af-198c5decffd4-kube-api-access-fntnn\") pod \"maas-api-e2e-worker-w4-825f30-5448495788-zbxs6\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:38.971908 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:38.971772    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f11e1b-28fb-4b8b-b6af-198c5decffd4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-825f30-5448495788-zbxs6\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:38.971908 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:38.971804    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f11e1b-28fb-4b8b-b6af-198c5decffd4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-825f30-5448495788-zbxs6\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:39.056541 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.056507    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:39.072947 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.072915    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fntnn\" (UniqueName: \"kubernetes.io/projected/71f11e1b-28fb-4b8b-b6af-198c5decffd4-kube-api-access-fntnn\") pod \"maas-api-e2e-worker-w4-825f30-5448495788-zbxs6\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:39.073075 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.072963    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f11e1b-28fb-4b8b-b6af-198c5decffd4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-825f30-5448495788-zbxs6\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:39.073075 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.073014    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f11e1b-28fb-4b8b-b6af-198c5decffd4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-825f30-5448495788-zbxs6\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:39.073162 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:18:39.073086    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-serving-cert: secret "maas-api-e2e-worker-w4-825f30-serving-cert" not found
Sep 17 23:18:39.073162 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:18:39.073153    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f11e1b-28fb-4b8b-b6af-198c5decffd4-maas-api-tls podName:71f11e1b-28fb-4b8b-b6af-198c5decffd4 nodeName:}" failed. No retries permitted until 2026-09-17 23:18:39.573137038 +0000 UTC m=+717.796812764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71f11e1b-28fb-4b8b-b6af-198c5decffd4-maas-api-tls") pod "maas-api-e2e-worker-w4-825f30-5448495788-zbxs6" (UID: "71f11e1b-28fb-4b8b-b6af-198c5decffd4") : secret "maas-api-e2e-worker-w4-825f30-serving-cert" not found
Sep 17 23:18:39.073266 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.073094    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f11e1b-28fb-4b8b-b6af-198c5decffd4-metrics-certs\") pod \"maas-api-e2e-worker-w4-825f30-5448495788-zbxs6\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:39.073720 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.073692    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f11e1b-28fb-4b8b-b6af-198c5decffd4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-825f30-5448495788-zbxs6\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:39.075571 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.075554    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f11e1b-28fb-4b8b-b6af-198c5decffd4-metrics-certs\") pod \"maas-api-e2e-worker-w4-825f30-5448495788-zbxs6\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:39.082074 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.082049    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fntnn\" (UniqueName: \"kubernetes.io/projected/71f11e1b-28fb-4b8b-b6af-198c5decffd4-kube-api-access-fntnn\") pod \"maas-api-e2e-worker-w4-825f30-5448495788-zbxs6\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:39.390278 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.390247    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:39.528754 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.528722    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:39.578183 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.578148    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f11e1b-28fb-4b8b-b6af-198c5decffd4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-825f30-5448495788-zbxs6\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:39.580716 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.580690    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f11e1b-28fb-4b8b-b6af-198c5decffd4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-825f30-5448495788-zbxs6\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:39.623698 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.623666    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:39.834373 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.834342    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:39.900193 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.900118    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"]
Sep 17 23:18:39.905311 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.905286    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:39.908056 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.908036    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-48d98f-serving-cert\""
Sep 17 23:18:39.917196 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.917164    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"]
Sep 17 23:18:39.939660 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.939625    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:39.981548 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.981512    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j646q\" (UniqueName: \"kubernetes.io/projected/63c8522c-fc35-48b2-b1a6-6fb35ca77784-kube-api-access-j646q\") pod \"maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:39.981727 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.981572    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-metrics-certs\") pod \"maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:39.981727 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.981606    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-maas-api-tls\") pod \"maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:39.981727 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:39.981652    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c8522c-fc35-48b2-b1a6-6fb35ca77784-openshift-service-ca\") pod \"maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:40.039783 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.039747    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:40.082755 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.082721    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-metrics-certs\") pod \"maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:40.082920 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.082767    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-maas-api-tls\") pod \"maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:40.082920 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.082820    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c8522c-fc35-48b2-b1a6-6fb35ca77784-openshift-service-ca\") pod \"maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:40.082920 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.082902    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j646q\" (UniqueName: \"kubernetes.io/projected/63c8522c-fc35-48b2-b1a6-6fb35ca77784-kube-api-access-j646q\") pod \"maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:40.083063 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:18:40.082945    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-serving-cert: secret "maas-api-e2e-inf-a-48d98f-serving-cert" not found
Sep 17 23:18:40.083063 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:18:40.083037    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-maas-api-tls podName:63c8522c-fc35-48b2-b1a6-6fb35ca77784 nodeName:}" failed. No retries permitted until 2026-09-17 23:18:40.583013956 +0000 UTC m=+718.806689701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-maas-api-tls") pod "maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk" (UID: "63c8522c-fc35-48b2-b1a6-6fb35ca77784") : secret "maas-api-e2e-inf-a-48d98f-serving-cert" not found
Sep 17 23:18:40.083620 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.083597    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c8522c-fc35-48b2-b1a6-6fb35ca77784-openshift-service-ca\") pod \"maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:40.085404 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.085356    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-metrics-certs\") pod \"maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:40.090594 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.090570    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j646q\" (UniqueName: \"kubernetes.io/projected/63c8522c-fc35-48b2-b1a6-6fb35ca77784-kube-api-access-j646q\") pod \"maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:40.107117 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.107090    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"]
Sep 17 23:18:40.109356 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:18:40.109327    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f11e1b_28fb_4b8b_b6af_198c5decffd4.slice/crio-4f8d029e55a72b19a19fb3a79d9459c2997822dd6b797891c5bef91c98c918f0 WatchSource:0}: Error finding container 4f8d029e55a72b19a19fb3a79d9459c2997822dd6b797891c5bef91c98c918f0: Status 404 returned error can't find the container with id 4f8d029e55a72b19a19fb3a79d9459c2997822dd6b797891c5bef91c98c918f0
Sep 17 23:18:40.110794 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.110768    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:18:40.315391 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.315352    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6" event={"ID":"71f11e1b-28fb-4b8b-b6af-198c5decffd4","Type":"ContainerStarted","Data":"4f8d029e55a72b19a19fb3a79d9459c2997822dd6b797891c5bef91c98c918f0"}
Sep 17 23:18:40.443554 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.443454    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:40.547248 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.547206    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:40.587176 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.587142    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-maas-api-tls\") pod \"maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:40.587358 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:18:40.587333    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-serving-cert: secret "maas-api-e2e-inf-a-48d98f-serving-cert" not found
Sep 17 23:18:40.587424 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:18:40.587411    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-maas-api-tls podName:63c8522c-fc35-48b2-b1a6-6fb35ca77784 nodeName:}" failed. No retries permitted until 2026-09-17 23:18:41.587394528 +0000 UTC m=+719.811070253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-maas-api-tls") pod "maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk" (UID: "63c8522c-fc35-48b2-b1a6-6fb35ca77784") : secret "maas-api-e2e-inf-a-48d98f-serving-cert" not found
Sep 17 23:18:40.671015 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.670982    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:40.904943 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:40.904911    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:41.320956 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.320912    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6" event={"ID":"71f11e1b-28fb-4b8b-b6af-198c5decffd4","Type":"ContainerStarted","Data":"1ee516df74bc9f506ba00cf516d5083264c317d1cb13cbff73508e32a44707ff"}
Sep 17 23:18:41.321124 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.321055    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:41.339421 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.339371    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6" podStartSLOduration=2.773405132 podStartE2EDuration="3.339353833s" totalImagesPullingTime="565.948701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:41.338421686 +0000 UTC m=+719.562097437" watchObservedRunningTime="2026-09-17 23:18:41.339353833 +0000 UTC m=+719.563029581"
Sep 17 23:18:41.596443 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.596351    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-maas-api-tls\") pod \"maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:41.598785 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.598754    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-maas-api-tls\") pod \"maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:41.687172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.687134    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"]
Sep 17 23:18:41.691245 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.691224    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:41.693694 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.693671    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-736809-serving-cert\""
Sep 17 23:18:41.700320 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.700297    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"]
Sep 17 23:18:41.719021 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.718996    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:41.725606 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.725583    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:41.798451 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.798417    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4hrn\" (UniqueName: \"kubernetes.io/projected/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-kube-api-access-x4hrn\") pod \"maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:41.798633 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.798529    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:41.798633 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.798559    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:41.798633 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.798600    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-metrics-certs\") pod \"maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:41.820728 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.820694    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:41.899816 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.899730    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:41.899816 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.899767    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:41.899816 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.899800    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-metrics-certs\") pod \"maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:41.900068 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.899844    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hrn\" (UniqueName: \"kubernetes.io/projected/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-kube-api-access-x4hrn\") pod \"maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:41.900558 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.900533    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:41.902298 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.902266    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-metrics-certs\") pod \"maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:41.902405 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.902374    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:41.907521 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.907504    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4hrn\" (UniqueName: \"kubernetes.io/projected/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-kube-api-access-x4hrn\") pod \"maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:41.927136 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.927104    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:41.993142 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:41.993108    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"]
Sep 17 23:18:41.995798 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:18:41.995768    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c8522c_fc35_48b2_b1a6_6fb35ca77784.slice/crio-ddc10bd5244cde40c3ace7cb8400a35972a326731271eeabe4e295d1011446d0 WatchSource:0}: Error finding container ddc10bd5244cde40c3ace7cb8400a35972a326731271eeabe4e295d1011446d0: Status 404 returned error can't find the container with id ddc10bd5244cde40c3ace7cb8400a35972a326731271eeabe4e295d1011446d0
Sep 17 23:18:42.003385 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:42.003357    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:42.189127 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:42.189048    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:42.275902 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:42.275862    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"]
Sep 17 23:18:42.277792 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:42.277772    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:42.278709 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:18:42.278688    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa69cff_a7ca_4d9a_ae45_9a9a1fc8e08a.slice/crio-f78313ec68a8c314d99c66277f12f6a32afe76fb30542d06f1d2577f0594f54f WatchSource:0}: Error finding container f78313ec68a8c314d99c66277f12f6a32afe76fb30542d06f1d2577f0594f54f: Status 404 returned error can't find the container with id f78313ec68a8c314d99c66277f12f6a32afe76fb30542d06f1d2577f0594f54f
Sep 17 23:18:42.328219 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:42.328178    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk" event={"ID":"63c8522c-fc35-48b2-b1a6-6fb35ca77784","Type":"ContainerStarted","Data":"ddc10bd5244cde40c3ace7cb8400a35972a326731271eeabe4e295d1011446d0"}
Sep 17 23:18:42.329605 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:42.329561    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk" event={"ID":"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a","Type":"ContainerStarted","Data":"f78313ec68a8c314d99c66277f12f6a32afe76fb30542d06f1d2577f0594f54f"}
Sep 17 23:18:42.634740 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:42.634709    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:43.099080 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:43.099030    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:43.198744 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:43.198709    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:43.341753 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:43.341718    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk" event={"ID":"63c8522c-fc35-48b2-b1a6-6fb35ca77784","Type":"ContainerStarted","Data":"eb4919f1122557de259fa7fdbc07fa0027aa6eb174f9fe4e4d148a7c32da111e"}
Sep 17 23:18:43.341993 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:43.341827    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:43.343182 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:43.343155    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:43.343542 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:43.343515    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk" event={"ID":"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a","Type":"ContainerStarted","Data":"b19c93b09400d67014dba5e6fdbc2c513d1afd6ab4b5c764748e5a6697761863"}
Sep 17 23:18:43.343668 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:43.343648    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:43.365149 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:43.365058    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk" podStartSLOduration=4.02324943 podStartE2EDuration="4.365040937s" totalImagesPullingTime="341.791507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:43.362296289 +0000 UTC m=+721.585972039" watchObservedRunningTime="2026-09-17 23:18:43.365040937 +0000 UTC m=+721.588716686"
Sep 17 23:18:43.387095 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:43.387043    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk" podStartSLOduration=2.033799932 podStartE2EDuration="2.387025856s" totalImagesPullingTime="353.225924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:43.385308148 +0000 UTC m=+721.608983933" watchObservedRunningTime="2026-09-17 23:18:43.387025856 +0000 UTC m=+721.610701604"
Sep 17 23:18:43.439608 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:43.439573    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:43.634813 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:43.634725    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:43.865803 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:43.865768    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:43.996677 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:43.996649    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:44.401537 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:44.401504    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:45.336590 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:45.336557    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:46.031143 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:46.031113    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:46.135070 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:46.135036    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:46.313454 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:46.313363    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:47.335501 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:47.335474    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:18:47.415604 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:47.415563    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:48.497075 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:48.497039    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:48.598539 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:48.598504    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:48.697922 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:48.697859    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:49.228710 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:49.228674    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:49.353811 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:49.353783    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:18:49.354206 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:49.354188    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:18:50.543946 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.543916    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"]
Sep 17 23:18:50.549204 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.549184    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:50.551486 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.551462    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-61e623-serving-cert\""
Sep 17 23:18:50.559050 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.559027    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"]
Sep 17 23:18:50.605387 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.605350    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:50.685298 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.685265    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf168f-32a7-4c0a-be24-69df43568155-metrics-certs\") pod \"maas-api-e2e-autores-61e623-786968fc8d-hwcdl\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:50.685461 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.685330    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cf168f-32a7-4c0a-be24-69df43568155-openshift-service-ca\") pod \"maas-api-e2e-autores-61e623-786968fc8d-hwcdl\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:50.685461 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.685406    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g29hp\" (UniqueName: \"kubernetes.io/projected/a8cf168f-32a7-4c0a-be24-69df43568155-kube-api-access-g29hp\") pod \"maas-api-e2e-autores-61e623-786968fc8d-hwcdl\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:50.685535 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.685478    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cf168f-32a7-4c0a-be24-69df43568155-maas-api-tls\") pod \"maas-api-e2e-autores-61e623-786968fc8d-hwcdl\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:50.703045 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.703017    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:50.786438 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.786397    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf168f-32a7-4c0a-be24-69df43568155-metrics-certs\") pod \"maas-api-e2e-autores-61e623-786968fc8d-hwcdl\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:50.786620 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.786466    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cf168f-32a7-4c0a-be24-69df43568155-openshift-service-ca\") pod \"maas-api-e2e-autores-61e623-786968fc8d-hwcdl\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:50.786620 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.786493    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g29hp\" (UniqueName: \"kubernetes.io/projected/a8cf168f-32a7-4c0a-be24-69df43568155-kube-api-access-g29hp\") pod \"maas-api-e2e-autores-61e623-786968fc8d-hwcdl\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:50.786620 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.786524    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cf168f-32a7-4c0a-be24-69df43568155-maas-api-tls\") pod \"maas-api-e2e-autores-61e623-786968fc8d-hwcdl\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:50.787180 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.787156    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cf168f-32a7-4c0a-be24-69df43568155-openshift-service-ca\") pod \"maas-api-e2e-autores-61e623-786968fc8d-hwcdl\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:50.788912 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.788886    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf168f-32a7-4c0a-be24-69df43568155-metrics-certs\") pod \"maas-api-e2e-autores-61e623-786968fc8d-hwcdl\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:50.789023 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.789006    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cf168f-32a7-4c0a-be24-69df43568155-maas-api-tls\") pod \"maas-api-e2e-autores-61e623-786968fc8d-hwcdl\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:50.795377 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.795317    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g29hp\" (UniqueName: \"kubernetes.io/projected/a8cf168f-32a7-4c0a-be24-69df43568155-kube-api-access-g29hp\") pod \"maas-api-e2e-autores-61e623-786968fc8d-hwcdl\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:50.803043 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.803004    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:50.859861 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:50.859827    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:51.130899 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:51.130859    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"]
Sep 17 23:18:51.133261 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:18:51.133215    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cf168f_32a7_4c0a_be24_69df43568155.slice/crio-e22a0b6b72806f2fa2698cd3e310bee2ab0381ace505b1b7b35b2f419213e598 WatchSource:0}: Error finding container e22a0b6b72806f2fa2698cd3e310bee2ab0381ace505b1b7b35b2f419213e598: Status 404 returned error can't find the container with id e22a0b6b72806f2fa2698cd3e310bee2ab0381ace505b1b7b35b2f419213e598
Sep 17 23:18:51.389106 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:51.389021    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl" event={"ID":"a8cf168f-32a7-4c0a-be24-69df43568155","Type":"ContainerStarted","Data":"e22a0b6b72806f2fa2698cd3e310bee2ab0381ace505b1b7b35b2f419213e598"}
Sep 17 23:18:51.437964 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:51.437930    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:51.570961 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:51.570913    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:52.395169 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:52.395131    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl" event={"ID":"a8cf168f-32a7-4c0a-be24-69df43568155","Type":"ContainerStarted","Data":"b64b134677add316bba1a4f70457926154341f6b7ee23577262c153020503c2e"}
Sep 17 23:18:52.395383 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:52.395190    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:52.420660 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:52.415193    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl" podStartSLOduration=2.091455649 podStartE2EDuration="2.415171909s" totalImagesPullingTime="323.71626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:52.413209218 +0000 UTC m=+730.636884966" watchObservedRunningTime="2026-09-17 23:18:52.415171909 +0000 UTC m=+730.638847658"
Sep 17 23:18:52.498016 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:52.497976    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:52.674298 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:52.674217    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:52.949231 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:52.949142    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:53.149199 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:53.149147    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:53.264695 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:53.264610    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:53.355742 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:53.355703    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:53.455737 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:53.455706    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:53.832401 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:53.832369    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:54.807957 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:54.807924    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"]
Sep 17 23:18:54.812066 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:54.812041    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:54.814256 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:54.814229    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-54be2504-serving-cert\""
Sep 17 23:18:54.822204 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:54.822180    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"]
Sep 17 23:18:54.848780 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:54.848754    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:54.923565 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:54.923537    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:54.931808 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:54.931782    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbb6251-0d9c-4b4d-95b3-943701de322b-maas-api-tls\") pod \"maas-api-e2e-ait-54be2504-78577c659c-x52hm\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:54.931933 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:54.931819    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbb6251-0d9c-4b4d-95b3-943701de322b-openshift-service-ca\") pod \"maas-api-e2e-ait-54be2504-78577c659c-x52hm\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:54.931933 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:54.931854    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t78xr\" (UniqueName: \"kubernetes.io/projected/7fbb6251-0d9c-4b4d-95b3-943701de322b-kube-api-access-t78xr\") pod \"maas-api-e2e-ait-54be2504-78577c659c-x52hm\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:54.932005 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:54.931975    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbb6251-0d9c-4b4d-95b3-943701de322b-metrics-certs\") pod \"maas-api-e2e-ait-54be2504-78577c659c-x52hm\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:55.032773 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:55.032735    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbb6251-0d9c-4b4d-95b3-943701de322b-metrics-certs\") pod \"maas-api-e2e-ait-54be2504-78577c659c-x52hm\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:55.033037 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:55.032801    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbb6251-0d9c-4b4d-95b3-943701de322b-maas-api-tls\") pod \"maas-api-e2e-ait-54be2504-78577c659c-x52hm\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:55.033037 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:55.032831    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbb6251-0d9c-4b4d-95b3-943701de322b-openshift-service-ca\") pod \"maas-api-e2e-ait-54be2504-78577c659c-x52hm\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:55.033037 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:55.032860    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78xr\" (UniqueName: \"kubernetes.io/projected/7fbb6251-0d9c-4b4d-95b3-943701de322b-kube-api-access-t78xr\") pod \"maas-api-e2e-ait-54be2504-78577c659c-x52hm\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:55.033037 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:18:55.032972    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-serving-cert: secret "maas-api-e2e-ait-54be2504-serving-cert" not found
Sep 17 23:18:55.033208 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:18:55.033046    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fbb6251-0d9c-4b4d-95b3-943701de322b-maas-api-tls podName:7fbb6251-0d9c-4b4d-95b3-943701de322b nodeName:}" failed. No retries permitted until 2026-09-17 23:18:55.533029121 +0000 UTC m=+733.756704846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7fbb6251-0d9c-4b4d-95b3-943701de322b-maas-api-tls") pod "maas-api-e2e-ait-54be2504-78577c659c-x52hm" (UID: "7fbb6251-0d9c-4b4d-95b3-943701de322b") : secret "maas-api-e2e-ait-54be2504-serving-cert" not found
Sep 17 23:18:55.033542 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:55.033523    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbb6251-0d9c-4b4d-95b3-943701de322b-openshift-service-ca\") pod \"maas-api-e2e-ait-54be2504-78577c659c-x52hm\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:55.035433 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:55.035399    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbb6251-0d9c-4b4d-95b3-943701de322b-metrics-certs\") pod \"maas-api-e2e-ait-54be2504-78577c659c-x52hm\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:55.040764 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:55.040746    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t78xr\" (UniqueName: \"kubernetes.io/projected/7fbb6251-0d9c-4b4d-95b3-943701de322b-kube-api-access-t78xr\") pod \"maas-api-e2e-ait-54be2504-78577c659c-x52hm\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:55.316111 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:55.316074    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:55.469690 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:55.469635    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:55.538399 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:55.538364    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbb6251-0d9c-4b4d-95b3-943701de322b-maas-api-tls\") pod \"maas-api-e2e-ait-54be2504-78577c659c-x52hm\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:55.540831 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:55.540811    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbb6251-0d9c-4b4d-95b3-943701de322b-maas-api-tls\") pod \"maas-api-e2e-ait-54be2504-78577c659c-x52hm\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:55.567236 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:55.567152    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:55.727357 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:55.727320    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:55.940270 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:55.940175    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:55.999482 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:55.999458    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"]
Sep 17 23:18:56.001819 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:18:56.001795    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbb6251_0d9c_4b4d_95b3_943701de322b.slice/crio-c38046ecf0db4ecb87c345d0ad51d5420f025fb75600f1c25efed637002c08b5 WatchSource:0}: Error finding container c38046ecf0db4ecb87c345d0ad51d5420f025fb75600f1c25efed637002c08b5: Status 404 returned error can't find the container with id c38046ecf0db4ecb87c345d0ad51d5420f025fb75600f1c25efed637002c08b5
Sep 17 23:18:56.162135 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:56.162096    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:56.415640 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:56.415616    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm" event={"ID":"7fbb6251-0d9c-4b4d-95b3-943701de322b","Type":"ContainerStarted","Data":"c38046ecf0db4ecb87c345d0ad51d5420f025fb75600f1c25efed637002c08b5"}
Sep 17 23:18:56.867792 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:56.867754    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:57.031981 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:57.031938    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:57.257390 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:57.257307    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:57.357568 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:57.357530    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:57.421718 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:57.421683    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm" event={"ID":"7fbb6251-0d9c-4b4d-95b3-943701de322b","Type":"ContainerStarted","Data":"14ea89830b213fb2408063c1d38769a8e5a321496f1a4600cbdba1ddb3f2bf70"}
Sep 17 23:18:57.421896 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:57.421759    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:18:57.442968 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:57.442921    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm" podStartSLOduration=3.03676727 podStartE2EDuration="3.442905181s" totalImagesPullingTime="406.137911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:57.440540603 +0000 UTC m=+735.664216352" watchObservedRunningTime="2026-09-17 23:18:57.442905181 +0000 UTC m=+735.666580929"
Sep 17 23:18:57.447079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:57.447055    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:57.516791 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:57.516708    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:57.892254 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:57.892224    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:58.406073 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:58.406043    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:18:59.166918 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:18:59.166850    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:00.087357 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:00.087320    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:00.711434 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:00.711404    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:01.657783 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:01.657753    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:01.991246 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:01.991160    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:02.290389 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:02.290356    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:02.611561 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:02.611465    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:03.016639 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:03.016531    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:03.116474 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:03.116443    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:03.434077 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:03.434047    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:19:05.082442 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:05.082401    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:06.477034 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:06.476995    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:06.575175 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:06.575142    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:06.732403 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:06.732324    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:07.852339 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:07.852195    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:08.711286 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.711251    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"]
Sep 17 23:19:08.716181 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.716158    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:08.718529 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.718504    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f38e68-serving-cert\""
Sep 17 23:19:08.726742 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.726717    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"]
Sep 17 23:19:08.751825 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.751791    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:08.852199 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.852166    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:08.869484 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.869451    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8387bbd7-2c95-4aea-94d7-2ab845e0e822-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:08.869920 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.869636    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl8dw\" (UniqueName: \"kubernetes.io/projected/8387bbd7-2c95-4aea-94d7-2ab845e0e822-kube-api-access-hl8dw\") pod \"maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:08.869920 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.869682    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8387bbd7-2c95-4aea-94d7-2ab845e0e822-maas-api-tls\") pod \"maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:08.869920 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.869718    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8387bbd7-2c95-4aea-94d7-2ab845e0e822-metrics-certs\") pod \"maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:08.910927 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.910857    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:08.970576 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.970482    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl8dw\" (UniqueName: \"kubernetes.io/projected/8387bbd7-2c95-4aea-94d7-2ab845e0e822-kube-api-access-hl8dw\") pod \"maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:08.970576 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.970522    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8387bbd7-2c95-4aea-94d7-2ab845e0e822-maas-api-tls\") pod \"maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:08.970576 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.970549    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8387bbd7-2c95-4aea-94d7-2ab845e0e822-metrics-certs\") pod \"maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:08.970909 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.970652    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8387bbd7-2c95-4aea-94d7-2ab845e0e822-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:08.971350 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.971328    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8387bbd7-2c95-4aea-94d7-2ab845e0e822-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:08.973052 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.973031    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8387bbd7-2c95-4aea-94d7-2ab845e0e822-metrics-certs\") pod \"maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:08.973144 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.973085    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8387bbd7-2c95-4aea-94d7-2ab845e0e822-maas-api-tls\") pod \"maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:08.978178 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:08.978156    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl8dw\" (UniqueName: \"kubernetes.io/projected/8387bbd7-2c95-4aea-94d7-2ab845e0e822-kube-api-access-hl8dw\") pod \"maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:09.028073 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:09.028039    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:09.302615 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:09.302577    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"]
Sep 17 23:19:09.305582 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:19:09.305548    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8387bbd7_2c95_4aea_94d7_2ab845e0e822.slice/crio-5a29ee7ad250e71c97dd2ae6c56c6b1a0bb4a87096a66931336dbd5616c35dc0 WatchSource:0}: Error finding container 5a29ee7ad250e71c97dd2ae6c56c6b1a0bb4a87096a66931336dbd5616c35dc0: Status 404 returned error can't find the container with id 5a29ee7ad250e71c97dd2ae6c56c6b1a0bb4a87096a66931336dbd5616c35dc0
Sep 17 23:19:09.325325 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:09.325297    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:09.424149 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:09.424118    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:09.483188 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:09.483153    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k" event={"ID":"8387bbd7-2c95-4aea-94d7-2ab845e0e822","Type":"ContainerStarted","Data":"5a29ee7ad250e71c97dd2ae6c56c6b1a0bb4a87096a66931336dbd5616c35dc0"}
Sep 17 23:19:09.767853 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:09.767826    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:10.489491 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:10.489457    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k" event={"ID":"8387bbd7-2c95-4aea-94d7-2ab845e0e822","Type":"ContainerStarted","Data":"7a53b3dbb7b3e78932777d29eabc0c73d2aa091c3c06c98cfe14957dbbd75c43"}
Sep 17 23:19:10.489895 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:10.489564    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:10.523683 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:10.523625    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k" podStartSLOduration=2.1233122 podStartE2EDuration="2.523607078s" totalImagesPullingTime="400.294878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:10.520890254 +0000 UTC m=+748.744565994" watchObservedRunningTime="2026-09-17 23:19:10.523607078 +0000 UTC m=+748.747282900"
Sep 17 23:19:11.017983 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:11.017951    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:11.951406 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:11.951372    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:12.578862 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:12.578830    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:12.678285 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:12.678248    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:12.810245 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:12.810215    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:12.911415 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:12.911313    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:13.734174 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:13.734144    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:14.751843 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:14.751796    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:15.860821 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:15.860779    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:16.499697 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:16.499667    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:19:16.649161 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:16.649131    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:17.374759 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:17.374728    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:18.338515 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:18.338476    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:19.011694 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:19.011663    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:19.572432 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:19.572398    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:19.671120 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:19.671090    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:19.767253 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:19.767220    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:19.927332 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:19.927250    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:20.027328 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:20.027288    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:20.862184 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:20.862155    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:21.654320 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:21.654283    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:22.452268 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:22.452227    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:23.363297 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:23.363256    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:24.070314 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:24.070278    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:24.824452 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:24.824417    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:25.737601 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:25.737568    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:25.998531 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:25.998445    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:26.098682 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:26.098647    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:26.201450 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:26.201412    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:26.697010 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:26.696970    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:29.728153 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:29.728116    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:30.280317 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.280283    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"]
Sep 17 23:19:30.280579 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.280524    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm" podUID="7fbb6251-0d9c-4b4d-95b3-943701de322b" containerName="maas-api" containerID="cri-o://14ea89830b213fb2408063c1d38769a8e5a321496f1a4600cbdba1ddb3f2bf70" gracePeriod=30
Sep 17 23:19:30.307751 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.307720    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:30.379348 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.379315    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:30.440858 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.440820    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:30.529792 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.529761    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:19:30.533428 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.533361    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:30.584787 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.584757    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbb6251-0d9c-4b4d-95b3-943701de322b-metrics-certs\") pod \"7fbb6251-0d9c-4b4d-95b3-943701de322b\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") "
Sep 17 23:19:30.584975 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.584815    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbb6251-0d9c-4b4d-95b3-943701de322b-maas-api-tls\") pod \"7fbb6251-0d9c-4b4d-95b3-943701de322b\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") "
Sep 17 23:19:30.584975 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.584840    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t78xr\" (UniqueName: \"kubernetes.io/projected/7fbb6251-0d9c-4b4d-95b3-943701de322b-kube-api-access-t78xr\") pod \"7fbb6251-0d9c-4b4d-95b3-943701de322b\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") "
Sep 17 23:19:30.584975 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.584935    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbb6251-0d9c-4b4d-95b3-943701de322b-openshift-service-ca\") pod \"7fbb6251-0d9c-4b4d-95b3-943701de322b\" (UID: \"7fbb6251-0d9c-4b4d-95b3-943701de322b\") "
Sep 17 23:19:30.585453 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.585407    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbb6251-0d9c-4b4d-95b3-943701de322b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7fbb6251-0d9c-4b4d-95b3-943701de322b" (UID: "7fbb6251-0d9c-4b4d-95b3-943701de322b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:19:30.587174 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.587145    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbb6251-0d9c-4b4d-95b3-943701de322b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7fbb6251-0d9c-4b4d-95b3-943701de322b" (UID: "7fbb6251-0d9c-4b4d-95b3-943701de322b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:19:30.587312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.587222    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbb6251-0d9c-4b4d-95b3-943701de322b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7fbb6251-0d9c-4b4d-95b3-943701de322b" (UID: "7fbb6251-0d9c-4b4d-95b3-943701de322b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:19:30.587312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.587270    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbb6251-0d9c-4b4d-95b3-943701de322b-kube-api-access-t78xr" (OuterVolumeSpecName: "kube-api-access-t78xr") pod "7fbb6251-0d9c-4b4d-95b3-943701de322b" (UID: "7fbb6251-0d9c-4b4d-95b3-943701de322b"). InnerVolumeSpecName "kube-api-access-t78xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:19:30.590217 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.590193    2568 generic.go:356] "Generic (PLEG): container finished" podID="7fbb6251-0d9c-4b4d-95b3-943701de322b" containerID="14ea89830b213fb2408063c1d38769a8e5a321496f1a4600cbdba1ddb3f2bf70" exitCode=0
Sep 17 23:19:30.590305 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.590233    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"
Sep 17 23:19:30.590305 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.590263    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm" event={"ID":"7fbb6251-0d9c-4b4d-95b3-943701de322b","Type":"ContainerDied","Data":"14ea89830b213fb2408063c1d38769a8e5a321496f1a4600cbdba1ddb3f2bf70"}
Sep 17 23:19:30.590377 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.590316    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm" event={"ID":"7fbb6251-0d9c-4b4d-95b3-943701de322b","Type":"ContainerDied","Data":"c38046ecf0db4ecb87c345d0ad51d5420f025fb75600f1c25efed637002c08b5"}
Sep 17 23:19:30.590377 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.590342    2568 scope.go:117] "RemoveContainer" containerID="14ea89830b213fb2408063c1d38769a8e5a321496f1a4600cbdba1ddb3f2bf70"
Sep 17 23:19:30.601429 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.601409    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:30.609031 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.609012    2568 scope.go:117] "RemoveContainer" containerID="14ea89830b213fb2408063c1d38769a8e5a321496f1a4600cbdba1ddb3f2bf70"
Sep 17 23:19:30.609349 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:19:30.609326    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14ea89830b213fb2408063c1d38769a8e5a321496f1a4600cbdba1ddb3f2bf70\": container with ID starting with 14ea89830b213fb2408063c1d38769a8e5a321496f1a4600cbdba1ddb3f2bf70 not found: ID does not exist" containerID="14ea89830b213fb2408063c1d38769a8e5a321496f1a4600cbdba1ddb3f2bf70"
Sep 17 23:19:30.609443 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.609357    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14ea89830b213fb2408063c1d38769a8e5a321496f1a4600cbdba1ddb3f2bf70"} err="failed to get container status \"14ea89830b213fb2408063c1d38769a8e5a321496f1a4600cbdba1ddb3f2bf70\": rpc error: code = NotFound desc = could not find container \"14ea89830b213fb2408063c1d38769a8e5a321496f1a4600cbdba1ddb3f2bf70\": container with ID starting with 14ea89830b213fb2408063c1d38769a8e5a321496f1a4600cbdba1ddb3f2bf70 not found: ID does not exist"
Sep 17 23:19:30.624088 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.624056    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"]
Sep 17 23:19:30.631490 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.631461    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54be2504-78577c659c-x52hm"]
Sep 17 23:19:30.686187 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.686157    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fbb6251-0d9c-4b4d-95b3-943701de322b-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:19:30.686187 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.686184    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fbb6251-0d9c-4b4d-95b3-943701de322b-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:19:30.686187 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.686195    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t78xr\" (UniqueName: \"kubernetes.io/projected/7fbb6251-0d9c-4b4d-95b3-943701de322b-kube-api-access-t78xr\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:19:30.686403 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.686207    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbb6251-0d9c-4b4d-95b3-943701de322b-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:19:30.694140 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:30.694099    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:31.130703 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:31.130639    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:31.235080 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:31.235039    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:31.343752 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:31.343720    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:31.557712 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:31.557671    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:31.779571 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:31.779498    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:31.882941 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:31.882840    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:31.990921 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:31.990857    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fbb6251-0d9c-4b4d-95b3-943701de322b" path="/var/lib/kubelet/pods/7fbb6251-0d9c-4b4d-95b3-943701de322b/volumes"
Sep 17 23:19:32.517342 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:32.517294    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:33.366603 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:33.366568    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:34.258859 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:34.258826    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:34.775610 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:34.775570    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:34.876431 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:34.876387    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:35.024116 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:35.024077    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:35.835837 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:35.835798    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"]
Sep 17 23:19:35.836472 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:35.836449    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbb6251-0d9c-4b4d-95b3-943701de322b" containerName="maas-api"
Sep 17 23:19:35.836530 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:35.836477    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbb6251-0d9c-4b4d-95b3-943701de322b" containerName="maas-api"
Sep 17 23:19:35.836618 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:35.836607    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbb6251-0d9c-4b4d-95b3-943701de322b" containerName="maas-api"
Sep 17 23:19:35.840474 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:35.840455    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:35.842775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:35.842754    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-eb3352f3-serving-cert\""
Sep 17 23:19:35.850582 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:35.850554    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"]
Sep 17 23:19:35.875014 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:35.874967    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:35.937740 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:35.937701    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e142579-c236-4c86-b3d3-e8e8427d497b-maas-api-tls\") pod \"maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:35.937740 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:35.937741    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e142579-c236-4c86-b3d3-e8e8427d497b-openshift-service-ca\") pod \"maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:35.937998 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:35.937853    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qdc6\" (UniqueName: \"kubernetes.io/projected/0e142579-c236-4c86-b3d3-e8e8427d497b-kube-api-access-9qdc6\") pod \"maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:35.937998 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:35.937949    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e142579-c236-4c86-b3d3-e8e8427d497b-metrics-certs\") pod \"maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:35.972199 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:35.972171    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:36.038923 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:36.038889    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e142579-c236-4c86-b3d3-e8e8427d497b-metrics-certs\") pod \"maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:36.039131 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:36.038950    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e142579-c236-4c86-b3d3-e8e8427d497b-maas-api-tls\") pod \"maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:36.039131 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:36.038969    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e142579-c236-4c86-b3d3-e8e8427d497b-openshift-service-ca\") pod \"maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:36.039131 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:36.039104    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qdc6\" (UniqueName: \"kubernetes.io/projected/0e142579-c236-4c86-b3d3-e8e8427d497b-kube-api-access-9qdc6\") pod \"maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:36.039768 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:36.039743    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e142579-c236-4c86-b3d3-e8e8427d497b-openshift-service-ca\") pod \"maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:36.041367 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:36.041335    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e142579-c236-4c86-b3d3-e8e8427d497b-maas-api-tls\") pod \"maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:36.041514 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:36.041496    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e142579-c236-4c86-b3d3-e8e8427d497b-metrics-certs\") pod \"maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:36.050437 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:36.050417    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qdc6\" (UniqueName: \"kubernetes.io/projected/0e142579-c236-4c86-b3d3-e8e8427d497b-kube-api-access-9qdc6\") pod \"maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:36.117600 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:36.117518    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:36.154129 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:36.154094    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:36.426097 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:36.426067    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"]
Sep 17 23:19:36.428457 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:19:36.428429    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e142579_c236_4c86_b3d3_e8e8427d497b.slice/crio-2d847e61df30985c40b6c4f32890c6cf4ab246997d17785dea6d121b2ae4c0e4 WatchSource:0}: Error finding container 2d847e61df30985c40b6c4f32890c6cf4ab246997d17785dea6d121b2ae4c0e4: Status 404 returned error can't find the container with id 2d847e61df30985c40b6c4f32890c6cf4ab246997d17785dea6d121b2ae4c0e4
Sep 17 23:19:36.481201 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:36.481169    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:36.577881 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:36.577842    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:36.622772 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:36.622738    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv" event={"ID":"0e142579-c236-4c86-b3d3-e8e8427d497b","Type":"ContainerStarted","Data":"2d847e61df30985c40b6c4f32890c6cf4ab246997d17785dea6d121b2ae4c0e4"}
Sep 17 23:19:36.942758 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:36.942716    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:37.629164 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:37.629125    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv" event={"ID":"0e142579-c236-4c86-b3d3-e8e8427d497b","Type":"ContainerStarted","Data":"046ed85cc14d9bd0e061daa82974f702f9992399c53b80fafbb74d28c387287b"}
Sep 17 23:19:37.629430 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:37.629220    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:37.646548 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:37.646497    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv" podStartSLOduration=2.2998801220000002 podStartE2EDuration="2.646478723s" totalImagesPullingTime="346.598601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:37.645853255 +0000 UTC m=+775.869529004" watchObservedRunningTime="2026-09-17 23:19:37.646478723 +0000 UTC m=+775.870154471"
Sep 17 23:19:37.756831 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:37.756797    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:38.455259 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:38.455223    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:39.230047 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:39.229997    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:39.330493 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:39.330458    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:39.935731 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:39.935699    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:43.640143 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:43.640115    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:19:58.030533 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:58.030445    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:58.990494 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:58.990460    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:59.737532 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:19:59.737487    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:00.470672 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:00.470642    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:01.320502 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:01.320465    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:02.004351 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:02.004303    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:02.800195 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:02.800145    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:03.663623 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:03.663581    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:04.349732 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:04.349699    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:05.040263 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:05.040234    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:06.045422 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:06.045386    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:06.696638 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:06.696603    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"]
Sep 17 23:20:06.696919 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:06.696852    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv" podUID="0e142579-c236-4c86-b3d3-e8e8427d497b" containerName="maas-api" containerID="cri-o://046ed85cc14d9bd0e061daa82974f702f9992399c53b80fafbb74d28c387287b" gracePeriod=30
Sep 17 23:20:06.724597 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:06.724569    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:06.791751 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:06.791720    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:06.848735 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:06.848697    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:06.925186 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:06.925158    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:06.943382 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:06.943359    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:20:06.989262 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:06.989183    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:07.044633 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.044600    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e142579-c236-4c86-b3d3-e8e8427d497b-maas-api-tls\") pod \"0e142579-c236-4c86-b3d3-e8e8427d497b\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") "
Sep 17 23:20:07.044813 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.044704    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e142579-c236-4c86-b3d3-e8e8427d497b-metrics-certs\") pod \"0e142579-c236-4c86-b3d3-e8e8427d497b\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") "
Sep 17 23:20:07.044813 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.044805    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e142579-c236-4c86-b3d3-e8e8427d497b-openshift-service-ca\") pod \"0e142579-c236-4c86-b3d3-e8e8427d497b\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") "
Sep 17 23:20:07.044967 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.044855    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qdc6\" (UniqueName: \"kubernetes.io/projected/0e142579-c236-4c86-b3d3-e8e8427d497b-kube-api-access-9qdc6\") pod \"0e142579-c236-4c86-b3d3-e8e8427d497b\" (UID: \"0e142579-c236-4c86-b3d3-e8e8427d497b\") "
Sep 17 23:20:07.045350 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.045162    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e142579-c236-4c86-b3d3-e8e8427d497b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e142579-c236-4c86-b3d3-e8e8427d497b" (UID: "0e142579-c236-4c86-b3d3-e8e8427d497b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:20:07.048055 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.048024    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e142579-c236-4c86-b3d3-e8e8427d497b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0e142579-c236-4c86-b3d3-e8e8427d497b" (UID: "0e142579-c236-4c86-b3d3-e8e8427d497b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:20:07.048378 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.048341    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e142579-c236-4c86-b3d3-e8e8427d497b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e142579-c236-4c86-b3d3-e8e8427d497b" (UID: "0e142579-c236-4c86-b3d3-e8e8427d497b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:20:07.048437 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.048380    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e142579-c236-4c86-b3d3-e8e8427d497b-kube-api-access-9qdc6" (OuterVolumeSpecName: "kube-api-access-9qdc6") pod "0e142579-c236-4c86-b3d3-e8e8427d497b" (UID: "0e142579-c236-4c86-b3d3-e8e8427d497b"). InnerVolumeSpecName "kube-api-access-9qdc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:20:07.082321 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.082287    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:07.145748 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.145708    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e142579-c236-4c86-b3d3-e8e8427d497b-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:20:07.145748 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.145740    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qdc6\" (UniqueName: \"kubernetes.io/projected/0e142579-c236-4c86-b3d3-e8e8427d497b-kube-api-access-9qdc6\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:20:07.145748 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.145752    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e142579-c236-4c86-b3d3-e8e8427d497b-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:20:07.146011 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.145760    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e142579-c236-4c86-b3d3-e8e8427d497b-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:20:07.334823 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.334793    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:07.509705 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.509665    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:07.781756 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.781726    2568 generic.go:356] "Generic (PLEG): container finished" podID="0e142579-c236-4c86-b3d3-e8e8427d497b" containerID="046ed85cc14d9bd0e061daa82974f702f9992399c53b80fafbb74d28c387287b" exitCode=0
Sep 17 23:20:07.781955 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.781792    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"
Sep 17 23:20:07.781955 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.781793    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv" event={"ID":"0e142579-c236-4c86-b3d3-e8e8427d497b","Type":"ContainerDied","Data":"046ed85cc14d9bd0e061daa82974f702f9992399c53b80fafbb74d28c387287b"}
Sep 17 23:20:07.781955 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.781936    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv" event={"ID":"0e142579-c236-4c86-b3d3-e8e8427d497b","Type":"ContainerDied","Data":"2d847e61df30985c40b6c4f32890c6cf4ab246997d17785dea6d121b2ae4c0e4"}
Sep 17 23:20:07.782089 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.781961    2568 scope.go:117] "RemoveContainer" containerID="046ed85cc14d9bd0e061daa82974f702f9992399c53b80fafbb74d28c387287b"
Sep 17 23:20:07.792006 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.791988    2568 scope.go:117] "RemoveContainer" containerID="046ed85cc14d9bd0e061daa82974f702f9992399c53b80fafbb74d28c387287b"
Sep 17 23:20:07.792321 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:20:07.792303    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046ed85cc14d9bd0e061daa82974f702f9992399c53b80fafbb74d28c387287b\": container with ID starting with 046ed85cc14d9bd0e061daa82974f702f9992399c53b80fafbb74d28c387287b not found: ID does not exist" containerID="046ed85cc14d9bd0e061daa82974f702f9992399c53b80fafbb74d28c387287b"
Sep 17 23:20:07.792437 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.792333    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046ed85cc14d9bd0e061daa82974f702f9992399c53b80fafbb74d28c387287b"} err="failed to get container status \"046ed85cc14d9bd0e061daa82974f702f9992399c53b80fafbb74d28c387287b\": rpc error: code = NotFound desc = could not find container \"046ed85cc14d9bd0e061daa82974f702f9992399c53b80fafbb74d28c387287b\": container with ID starting with 046ed85cc14d9bd0e061daa82974f702f9992399c53b80fafbb74d28c387287b not found: ID does not exist"
Sep 17 23:20:07.806559 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.806532    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"]
Sep 17 23:20:07.812556 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.812528    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eb3352f3-689cdfb948-svrsv"]
Sep 17 23:20:07.818241 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.818218    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:07.990310 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:07.990274    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e142579-c236-4c86-b3d3-e8e8427d497b" path="/var/lib/kubelet/pods/0e142579-c236-4c86-b3d3-e8e8427d497b/volumes"
Sep 17 23:20:08.105507 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:08.105414    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:08.239274 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:08.239232    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:08.861581 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:08.861546    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:09.741802 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:09.741768    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:10.058140 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:10.058102    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:10.156205 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:10.156165    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:10.784741 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:10.784700    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:11.624451 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:11.624417    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:12.298190 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.298156    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"]
Sep 17 23:20:12.298581 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.298559    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e142579-c236-4c86-b3d3-e8e8427d497b" containerName="maas-api"
Sep 17 23:20:12.298581 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.298570    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e142579-c236-4c86-b3d3-e8e8427d497b" containerName="maas-api"
Sep 17 23:20:12.298669 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.298637    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e142579-c236-4c86-b3d3-e8e8427d497b" containerName="maas-api"
Sep 17 23:20:12.303251 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.303226    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:12.306081 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.306009    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5ab8e6b6-serving-cert\""
Sep 17 23:20:12.313466 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.313439    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"]
Sep 17 23:20:12.334187 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.334149    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:12.399161 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.399124    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc5jl\" (UniqueName: \"kubernetes.io/projected/3f391ef6-e786-4ded-9617-1e8b57c3ed43-kube-api-access-zc5jl\") pod \"maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:12.399338 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.399167    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f391ef6-e786-4ded-9617-1e8b57c3ed43-metrics-certs\") pod \"maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:12.399338 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.399229    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f391ef6-e786-4ded-9617-1e8b57c3ed43-openshift-service-ca\") pod \"maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:12.399338 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.399307    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f391ef6-e786-4ded-9617-1e8b57c3ed43-maas-api-tls\") pod \"maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:12.434445 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.434411    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:12.500082 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.500039    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f391ef6-e786-4ded-9617-1e8b57c3ed43-metrics-certs\") pod \"maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:12.500285 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.500090    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f391ef6-e786-4ded-9617-1e8b57c3ed43-openshift-service-ca\") pod \"maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:12.500285 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.500166    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f391ef6-e786-4ded-9617-1e8b57c3ed43-maas-api-tls\") pod \"maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:12.500285 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.500275    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc5jl\" (UniqueName: \"kubernetes.io/projected/3f391ef6-e786-4ded-9617-1e8b57c3ed43-kube-api-access-zc5jl\") pod \"maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:12.500460 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:20:12.500335    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-serving-cert: secret "maas-api-e2e-ait-5ab8e6b6-serving-cert" not found
Sep 17 23:20:12.500460 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:20:12.500416    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f391ef6-e786-4ded-9617-1e8b57c3ed43-maas-api-tls podName:3f391ef6-e786-4ded-9617-1e8b57c3ed43 nodeName:}" failed. No retries permitted until 2026-09-17 23:20:13.000393706 +0000 UTC m=+811.224069447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f391ef6-e786-4ded-9617-1e8b57c3ed43-maas-api-tls") pod "maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd" (UID: "3f391ef6-e786-4ded-9617-1e8b57c3ed43") : secret "maas-api-e2e-ait-5ab8e6b6-serving-cert" not found
Sep 17 23:20:12.500957 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.500938    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f391ef6-e786-4ded-9617-1e8b57c3ed43-openshift-service-ca\") pod \"maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:12.502577 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.502560    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f391ef6-e786-4ded-9617-1e8b57c3ed43-metrics-certs\") pod \"maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:12.513127 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.513099    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc5jl\" (UniqueName: \"kubernetes.io/projected/3f391ef6-e786-4ded-9617-1e8b57c3ed43-kube-api-access-zc5jl\") pod \"maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:12.620477 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.620375    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:12.921561 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:12.921435    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:13.005020 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:13.004981    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f391ef6-e786-4ded-9617-1e8b57c3ed43-maas-api-tls\") pod \"maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:13.007482 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:13.007454    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f391ef6-e786-4ded-9617-1e8b57c3ed43-maas-api-tls\") pod \"maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:13.021639 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:13.021609    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:13.216580 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:13.216491    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:13.356548 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:13.356514    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:13.490892 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:13.490693    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"]
Sep 17 23:20:13.493140 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:20:13.493119    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f391ef6_e786_4ded_9617_1e8b57c3ed43.slice/crio-a8a6fec5d95486f020a65f5e026b90e2140510ab3a556c2002f47f452f218fb9 WatchSource:0}: Error finding container a8a6fec5d95486f020a65f5e026b90e2140510ab3a556c2002f47f452f218fb9: Status 404 returned error can't find the container with id a8a6fec5d95486f020a65f5e026b90e2140510ab3a556c2002f47f452f218fb9
Sep 17 23:20:13.814153 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:13.814113    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd" event={"ID":"3f391ef6-e786-4ded-9617-1e8b57c3ed43","Type":"ContainerStarted","Data":"a8a6fec5d95486f020a65f5e026b90e2140510ab3a556c2002f47f452f218fb9"}
Sep 17 23:20:14.351042 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:14.351001    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:14.820939 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:14.820900    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd" event={"ID":"3f391ef6-e786-4ded-9617-1e8b57c3ed43","Type":"ContainerStarted","Data":"5ca14c83829ccc400272066af36ab8528a38663f3b591e6f917660e772a83159"}
Sep 17 23:20:14.821323 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:14.820968    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:14.842784 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:14.842727    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd" podStartSLOduration=2.5003019650000002 podStartE2EDuration="2.842711133s" totalImagesPullingTime="342.409168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:14.840861471 +0000 UTC m=+813.064537222" watchObservedRunningTime="2026-09-17 23:20:14.842711133 +0000 UTC m=+813.066386921"
Sep 17 23:20:15.053683 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:15.053645    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:15.839778 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:15.839741    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:20.832931 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:20.832900    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:21.885680 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:21.885639    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"]
Sep 17 23:20:21.890291 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:21.890268    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:21.892571 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:21.892546    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-825f30\"/\"kube-root-ca.crt\""
Sep 17 23:20:21.892709 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:21.892583    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-825f30\"/\"openshift-service-ca.crt\""
Sep 17 23:20:21.893684 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:21.893667    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-825f30\"/\"simulator-w4-825f30-kserve-self-signed-certs\""
Sep 17 23:20:21.893754 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:21.893704    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-825f30\"/\"default-dockercfg-v6k5c\""
Sep 17 23:20:21.897947 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:21.897922    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"]
Sep 17 23:20:21.909333 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:21.909299    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:21.995276 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:21.995241    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-tmp-dir\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:21.995276 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:21.995281    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-dshm\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:21.995480 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:21.995311    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-model-cache\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:21.995480 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:21.995354    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2brp\" (UniqueName: \"kubernetes.io/projected/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-kube-api-access-r2brp\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:21.995480 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:21.995403    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-tls-certs\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:21.995480 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:21.995424    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-home\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:22.096658 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.096615    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-tmp-dir\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:22.096658 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.096667    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-dshm\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:22.096964 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.096696    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-model-cache\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:22.096964 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.096712    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2brp\" (UniqueName: \"kubernetes.io/projected/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-kube-api-access-r2brp\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:22.096964 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.096740    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-tls-certs\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:22.096964 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.096756    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-home\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:22.097173 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.097129    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-tmp-dir\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:22.097173 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.097164    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-home\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:22.097274 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.097189    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-model-cache\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:22.099189 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.099159    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-dshm\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:22.099479 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.099454    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-tls-certs\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:22.104987 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.104962    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2brp\" (UniqueName: \"kubernetes.io/projected/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-kube-api-access-r2brp\") pod \"simulator-w4-825f30-kserve-648c59db6c-2kwzs\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") " pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:22.201358 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.201257    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:22.473127 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.473098    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"]
Sep 17 23:20:22.475115 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:20:22.475080    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3028dec1_b28d_4853_a9fb_ab9578fbc4ba.slice/crio-5803869c092dd7b3de44e7e1b2075ed18545ecfc6546d21086ca6582c8a16b47 WatchSource:0}: Error finding container 5803869c092dd7b3de44e7e1b2075ed18545ecfc6546d21086ca6582c8a16b47: Status 404 returned error can't find the container with id 5803869c092dd7b3de44e7e1b2075ed18545ecfc6546d21086ca6582c8a16b47
Sep 17 23:20:22.865857 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.865819    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs" event={"ID":"3028dec1-b28d-4853-a9fb-ab9578fbc4ba","Type":"ContainerStarted","Data":"18d4d9bfc6b3e3b517501c169e0dda4e30b0fa310d66d906d0f3d4b0824c5d40"}
Sep 17 23:20:22.866038 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.865880    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs" event={"ID":"3028dec1-b28d-4853-a9fb-ab9578fbc4ba","Type":"ContainerStarted","Data":"5803869c092dd7b3de44e7e1b2075ed18545ecfc6546d21086ca6582c8a16b47"}
Sep 17 23:20:22.882527 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:22.882475    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs" podStartSLOduration=1.8824571300000001 podStartE2EDuration="1.88245713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:22.882172678 +0000 UTC m=+821.105848430" watchObservedRunningTime="2026-09-17 23:20:22.88245713 +0000 UTC m=+821.106132894"
Sep 17 23:20:32.202294 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:32.202257    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:32.202815 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:32.202306    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:32.214966 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:32.214936    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:32.932038 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:32.932001    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:20:34.533219 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:34.533186    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:35.341020 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:35.340986    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:36.140612 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:36.140557    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:36.928705 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:36.928660    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:37.841894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:37.841831    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:38.245478 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:38.245386    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:38.528620 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:38.528580    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:39.431896 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:39.431837    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:40.083384 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:40.083350    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:40.750143 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:40.750096    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:41.535585 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:41.535546    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:42.072681 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.072650    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"]
Sep 17 23:20:42.073288 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.072853    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd" podUID="3f391ef6-e786-4ded-9617-1e8b57c3ed43" containerName="maas-api" containerID="cri-o://5ca14c83829ccc400272066af36ab8528a38663f3b591e6f917660e772a83159" gracePeriod=30
Sep 17 23:20:42.101532 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.101506    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:42.171656 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.171627    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:42.225097 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.225066    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:42.292226 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.292196    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:42.326271 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.326203    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:42.349061 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.349026    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:42.402612 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.402578    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f391ef6-e786-4ded-9617-1e8b57c3ed43-metrics-certs\") pod \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") "
Sep 17 23:20:42.402789 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.402626    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc5jl\" (UniqueName: \"kubernetes.io/projected/3f391ef6-e786-4ded-9617-1e8b57c3ed43-kube-api-access-zc5jl\") pod \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") "
Sep 17 23:20:42.402789 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.402711    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f391ef6-e786-4ded-9617-1e8b57c3ed43-openshift-service-ca\") pod \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") "
Sep 17 23:20:42.402789 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.402751    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f391ef6-e786-4ded-9617-1e8b57c3ed43-maas-api-tls\") pod \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\" (UID: \"3f391ef6-e786-4ded-9617-1e8b57c3ed43\") "
Sep 17 23:20:42.403157 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.403131    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f391ef6-e786-4ded-9617-1e8b57c3ed43-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f391ef6-e786-4ded-9617-1e8b57c3ed43" (UID: "3f391ef6-e786-4ded-9617-1e8b57c3ed43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:20:42.404823 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.404798    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f391ef6-e786-4ded-9617-1e8b57c3ed43-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f391ef6-e786-4ded-9617-1e8b57c3ed43" (UID: "3f391ef6-e786-4ded-9617-1e8b57c3ed43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:20:42.404941 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.404823    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f391ef6-e786-4ded-9617-1e8b57c3ed43-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3f391ef6-e786-4ded-9617-1e8b57c3ed43" (UID: "3f391ef6-e786-4ded-9617-1e8b57c3ed43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:20:42.404941 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.404839    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f391ef6-e786-4ded-9617-1e8b57c3ed43-kube-api-access-zc5jl" (OuterVolumeSpecName: "kube-api-access-zc5jl") pod "3f391ef6-e786-4ded-9617-1e8b57c3ed43" (UID: "3f391ef6-e786-4ded-9617-1e8b57c3ed43"). InnerVolumeSpecName "kube-api-access-zc5jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:20:42.504576 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.504527    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f391ef6-e786-4ded-9617-1e8b57c3ed43-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:20:42.504576 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.504564    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f391ef6-e786-4ded-9617-1e8b57c3ed43-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:20:42.504576 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.504574    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc5jl\" (UniqueName: \"kubernetes.io/projected/3f391ef6-e786-4ded-9617-1e8b57c3ed43-kube-api-access-zc5jl\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:20:42.504576 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.504583    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f391ef6-e786-4ded-9617-1e8b57c3ed43-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:20:42.760065 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.759984    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:42.969951 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.969916    2568 generic.go:356] "Generic (PLEG): container finished" podID="3f391ef6-e786-4ded-9617-1e8b57c3ed43" containerID="5ca14c83829ccc400272066af36ab8528a38663f3b591e6f917660e772a83159" exitCode=0
Sep 17 23:20:42.970143 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.969964    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"
Sep 17 23:20:42.970143 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.969988    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd" event={"ID":"3f391ef6-e786-4ded-9617-1e8b57c3ed43","Type":"ContainerDied","Data":"5ca14c83829ccc400272066af36ab8528a38663f3b591e6f917660e772a83159"}
Sep 17 23:20:42.970143 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.970045    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd" event={"ID":"3f391ef6-e786-4ded-9617-1e8b57c3ed43","Type":"ContainerDied","Data":"a8a6fec5d95486f020a65f5e026b90e2140510ab3a556c2002f47f452f218fb9"}
Sep 17 23:20:42.970143 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.970071    2568 scope.go:117] "RemoveContainer" containerID="5ca14c83829ccc400272066af36ab8528a38663f3b591e6f917660e772a83159"
Sep 17 23:20:42.979975 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.979955    2568 scope.go:117] "RemoveContainer" containerID="5ca14c83829ccc400272066af36ab8528a38663f3b591e6f917660e772a83159"
Sep 17 23:20:42.980283 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:20:42.980264    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca14c83829ccc400272066af36ab8528a38663f3b591e6f917660e772a83159\": container with ID starting with 5ca14c83829ccc400272066af36ab8528a38663f3b591e6f917660e772a83159 not found: ID does not exist" containerID="5ca14c83829ccc400272066af36ab8528a38663f3b591e6f917660e772a83159"
Sep 17 23:20:42.980330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.980292    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca14c83829ccc400272066af36ab8528a38663f3b591e6f917660e772a83159"} err="failed to get container status \"5ca14c83829ccc400272066af36ab8528a38663f3b591e6f917660e772a83159\": rpc error: code = NotFound desc = could not find container \"5ca14c83829ccc400272066af36ab8528a38663f3b591e6f917660e772a83159\": container with ID starting with 5ca14c83829ccc400272066af36ab8528a38663f3b591e6f917660e772a83159 not found: ID does not exist"
Sep 17 23:20:42.990690 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.990664    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"]
Sep 17 23:20:42.996554 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:42.996525    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ab8e6b6-654986db5f-gpcqd"]
Sep 17 23:20:43.407117 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:43.407087    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:43.507167 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:43.507140    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:43.608196 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:43.608154    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:43.709691 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:43.709602    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:43.891120 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:43.891087    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:43.991718 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:43.991685    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f391ef6-e786-4ded-9617-1e8b57c3ed43" path="/var/lib/kubelet/pods/3f391ef6-e786-4ded-9617-1e8b57c3ed43/volumes"
Sep 17 23:20:44.143698 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:44.143663    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:44.366529 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:44.366494    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:44.580928 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:44.580892    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:45.326742 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:45.326710    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:46.039586 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:46.039553    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:46.825305 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:46.825266    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:47.208034 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:47.207944    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:47.310995 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:47.310953    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:47.411793 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:47.411761    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:47.831883 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:47.831841    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:48.554041 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:48.554005    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:48.656143 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:48.656111    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:49.439447 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:49.439414    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:50.091888 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.091842    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"]
Sep 17 23:20:50.092429 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.092412    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f391ef6-e786-4ded-9617-1e8b57c3ed43" containerName="maas-api"
Sep 17 23:20:50.092429 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.092430    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f391ef6-e786-4ded-9617-1e8b57c3ed43" containerName="maas-api"
Sep 17 23:20:50.092548 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.092498    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f391ef6-e786-4ded-9617-1e8b57c3ed43" containerName="maas-api"
Sep 17 23:20:50.097082 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.097062    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.100569 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.100542    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-48d98f\"/\"kube-root-ca.crt\""
Sep 17 23:20:50.100834 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.100812    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-48d98f\"/\"default-dockercfg-5vbwx\""
Sep 17 23:20:50.101280 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.101262    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-48d98f\"/\"openshift-service-ca.crt\""
Sep 17 23:20:50.101409 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.101355    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-48d98f\"/\"test-model-48d98f-kserve-self-signed-certs\""
Sep 17 23:20:50.103332 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.103287    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"]
Sep 17 23:20:50.112257 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.112231    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:50.177484 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.177453    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-home\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.177484 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.177490    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-dshm\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.177695 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.177509    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-model-cache\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.177695 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.177537    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-tmp-dir\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.177695 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.177592    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sd97\" (UniqueName: \"kubernetes.io/projected/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-kube-api-access-5sd97\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.177797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.177709    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-tls-certs\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.241249 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.241219    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:50.278643 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.278607    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-tls-certs\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.278819 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.278667    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-home\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.278819 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.278691    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-dshm\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.278819 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.278709    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-model-cache\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.278819 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.278727    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-tmp-dir\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.278819 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.278754    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sd97\" (UniqueName: \"kubernetes.io/projected/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-kube-api-access-5sd97\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.279179 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.279159    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-home\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.279254 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.279190    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-tmp-dir\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.279254 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.279214    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-model-cache\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.281008 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.280989    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-dshm\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.281297 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.281279    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-tls-certs\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.286128 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.286106    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sd97\" (UniqueName: \"kubernetes.io/projected/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-kube-api-access-5sd97\") pod \"test-model-48d98f-kserve-b6dc85d99-bz7ld\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") " pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.407228 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.407139    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:20:50.677909 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.677881    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"]
Sep 17 23:20:50.680606 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:20:50.680577    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod667a1ddb_2e75_45ae_8f61_9ab806b2b6fc.slice/crio-7490fb11c225f2a27212af43b515a3368f9a87baf9ab659681657793e8a15757 WatchSource:0}: Error finding container 7490fb11c225f2a27212af43b515a3368f9a87baf9ab659681657793e8a15757: Status 404 returned error can't find the container with id 7490fb11c225f2a27212af43b515a3368f9a87baf9ab659681657793e8a15757
Sep 17 23:20:50.923844 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:50.923810    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:51.010139 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.010057    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld" event={"ID":"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc","Type":"ContainerStarted","Data":"2e7c744dad0586ee2706e7facfdcdea2139d9add194b4a5c2ee824e9599c2d8c"}
Sep 17 23:20:51.010139 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.010098    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld" event={"ID":"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc","Type":"ContainerStarted","Data":"7490fb11c225f2a27212af43b515a3368f9a87baf9ab659681657793e8a15757"}
Sep 17 23:20:51.029445 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.029385    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld" podStartSLOduration=1.029369923 podStartE2EDuration="1.029369923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:51.028516351 +0000 UTC m=+849.252192093" watchObservedRunningTime="2026-09-17 23:20:51.029369923 +0000 UTC m=+849.253045671"
Sep 17 23:20:51.556402 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.556368    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"]
Sep 17 23:20:51.559735 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.559713    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:20:51.562311 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.562278    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-46ccf8cf-serving-cert\""
Sep 17 23:20:51.570678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.570654    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"]
Sep 17 23:20:51.596211 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.596183    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:51.691234 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.691198    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8ztw\" (UniqueName: \"kubernetes.io/projected/102402db-1d69-45d5-ae13-87c03aa62f89-kube-api-access-m8ztw\") pod \"maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:20:51.691234 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.691236    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/102402db-1d69-45d5-ae13-87c03aa62f89-openshift-service-ca\") pod \"maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:20:51.691475 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.691271    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102402db-1d69-45d5-ae13-87c03aa62f89-metrics-certs\") pod \"maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:20:51.691475 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.691333    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102402db-1d69-45d5-ae13-87c03aa62f89-maas-api-tls\") pod \"maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:20:51.696651 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.696622    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:51.792843 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.792804    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ztw\" (UniqueName: \"kubernetes.io/projected/102402db-1d69-45d5-ae13-87c03aa62f89-kube-api-access-m8ztw\") pod \"maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:20:51.792843 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.792851    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/102402db-1d69-45d5-ae13-87c03aa62f89-openshift-service-ca\") pod \"maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:20:51.793160 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.792918    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102402db-1d69-45d5-ae13-87c03aa62f89-metrics-certs\") pod \"maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:20:51.793160 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.792960    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102402db-1d69-45d5-ae13-87c03aa62f89-maas-api-tls\") pod \"maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:20:51.793601 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.793577    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/102402db-1d69-45d5-ae13-87c03aa62f89-openshift-service-ca\") pod \"maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:20:51.795595 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.795546    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102402db-1d69-45d5-ae13-87c03aa62f89-maas-api-tls\") pod \"maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:20:51.795686 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.795590    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102402db-1d69-45d5-ae13-87c03aa62f89-metrics-certs\") pod \"maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:20:51.796116 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.796097    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:51.801355 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.801332    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ztw\" (UniqueName: \"kubernetes.io/projected/102402db-1d69-45d5-ae13-87c03aa62f89-kube-api-access-m8ztw\") pod \"maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:20:51.872659 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:51.872565    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:20:52.073310 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:52.073276    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:52.148752 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:52.148669    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"]
Sep 17 23:20:52.150393 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:52.150354    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:52.152431 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:20:52.152408    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102402db_1d69_45d5_ae13_87c03aa62f89.slice/crio-0cd1b479b3dafa116ef2e331a5487aed3f2e11b9d83a58bf8e57e23ae408bf00 WatchSource:0}: Error finding container 0cd1b479b3dafa116ef2e331a5487aed3f2e11b9d83a58bf8e57e23ae408bf00: Status 404 returned error can't find the container with id 0cd1b479b3dafa116ef2e331a5487aed3f2e11b9d83a58bf8e57e23ae408bf00
Sep 17 23:20:53.021692 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:53.021659    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk" event={"ID":"102402db-1d69-45d5-ae13-87c03aa62f89","Type":"ContainerStarted","Data":"9f35b3deaa63fdd2e7992dff6404aca53ac759b04220a46e2c29d47cb9414c09"}
Sep 17 23:20:53.022198 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:53.021710    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk" event={"ID":"102402db-1d69-45d5-ae13-87c03aa62f89","Type":"ContainerStarted","Data":"0cd1b479b3dafa116ef2e331a5487aed3f2e11b9d83a58bf8e57e23ae408bf00"}
Sep 17 23:20:53.022198 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:53.021744    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:20:53.040100 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:53.040051    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk" podStartSLOduration=1.697401464 podStartE2EDuration="2.040036144s" totalImagesPullingTime="342.63468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:20:53.037717939 +0000 UTC m=+851.261393686" watchObservedRunningTime="2026-09-17 23:20:53.040036144 +0000 UTC m=+851.263711892"
Sep 17 23:20:56.973429 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:56.973397    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:20:59.032170 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:20:59.032138    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:21:00.407559 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:00.407493    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:21:00.407559 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:00.407562    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:21:00.420411 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:00.420376    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:21:01.071965 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:01.071931    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:21:09.421885 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:09.421838    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:10.167090 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:10.167040    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:10.945279 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:10.945245    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:11.600715 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:11.600680    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:12.245222 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:12.245186    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:13.229223 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:13.229184    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:14.140990 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:14.140960    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:14.825822 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:14.825784    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:15.466437 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:15.466403    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:16.324855 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:16.324816    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:17.087940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:17.087909    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:17.832273 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:17.832236    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:18.701213 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:18.701171    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:19.351115 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:19.351083    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:20.022309 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:20.022274    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:20.841701 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:20.841658    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:21.398274 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.398239    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"]
Sep 17 23:21:21.398735 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.398446    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl" podUID="a8cf168f-32a7-4c0a-be24-69df43568155" containerName="maas-api" containerID="cri-o://b64b134677add316bba1a4f70457926154341f6b7ee23577262c153020503c2e" gracePeriod=30
Sep 17 23:21:21.413368 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.413334    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:21.479453 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.479422    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:21.538397 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.538361    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:21.621739 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.621708    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:21.658406 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.658337    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:21:21.687147 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.687121    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:21.755494 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.755456    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:21.804767 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.804737    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cf168f-32a7-4c0a-be24-69df43568155-maas-api-tls\") pod \"a8cf168f-32a7-4c0a-be24-69df43568155\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") "
Sep 17 23:21:21.804967 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.804803    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cf168f-32a7-4c0a-be24-69df43568155-openshift-service-ca\") pod \"a8cf168f-32a7-4c0a-be24-69df43568155\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") "
Sep 17 23:21:21.804967 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.804844    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g29hp\" (UniqueName: \"kubernetes.io/projected/a8cf168f-32a7-4c0a-be24-69df43568155-kube-api-access-g29hp\") pod \"a8cf168f-32a7-4c0a-be24-69df43568155\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") "
Sep 17 23:21:21.804967 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.804958    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf168f-32a7-4c0a-be24-69df43568155-metrics-certs\") pod \"a8cf168f-32a7-4c0a-be24-69df43568155\" (UID: \"a8cf168f-32a7-4c0a-be24-69df43568155\") "
Sep 17 23:21:21.805221 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.805196    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8cf168f-32a7-4c0a-be24-69df43568155-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a8cf168f-32a7-4c0a-be24-69df43568155" (UID: "a8cf168f-32a7-4c0a-be24-69df43568155"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:21:21.805318 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.805304    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cf168f-32a7-4c0a-be24-69df43568155-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:21:21.807092 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.807065    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cf168f-32a7-4c0a-be24-69df43568155-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a8cf168f-32a7-4c0a-be24-69df43568155" (UID: "a8cf168f-32a7-4c0a-be24-69df43568155"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:21:21.807092 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.807082    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cf168f-32a7-4c0a-be24-69df43568155-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a8cf168f-32a7-4c0a-be24-69df43568155" (UID: "a8cf168f-32a7-4c0a-be24-69df43568155"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:21:21.807235 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.807108    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cf168f-32a7-4c0a-be24-69df43568155-kube-api-access-g29hp" (OuterVolumeSpecName: "kube-api-access-g29hp") pod "a8cf168f-32a7-4c0a-be24-69df43568155" (UID: "a8cf168f-32a7-4c0a-be24-69df43568155"). InnerVolumeSpecName "kube-api-access-g29hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:21:21.906118 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.906082    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cf168f-32a7-4c0a-be24-69df43568155-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:21:21.906118 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.906114    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g29hp\" (UniqueName: \"kubernetes.io/projected/a8cf168f-32a7-4c0a-be24-69df43568155-kube-api-access-g29hp\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:21:21.906118 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.906126    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cf168f-32a7-4c0a-be24-69df43568155-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:21:21.956532 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:21.956425    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:22.168337 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:22.168304    2568 generic.go:356] "Generic (PLEG): container finished" podID="a8cf168f-32a7-4c0a-be24-69df43568155" containerID="b64b134677add316bba1a4f70457926154341f6b7ee23577262c153020503c2e" exitCode=0
Sep 17 23:21:22.168498 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:22.168353    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"
Sep 17 23:21:22.168498 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:22.168385    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl" event={"ID":"a8cf168f-32a7-4c0a-be24-69df43568155","Type":"ContainerDied","Data":"b64b134677add316bba1a4f70457926154341f6b7ee23577262c153020503c2e"}
Sep 17 23:21:22.168498 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:22.168444    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl" event={"ID":"a8cf168f-32a7-4c0a-be24-69df43568155","Type":"ContainerDied","Data":"e22a0b6b72806f2fa2698cd3e310bee2ab0381ace505b1b7b35b2f419213e598"}
Sep 17 23:21:22.168498 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:22.168474    2568 scope.go:117] "RemoveContainer" containerID="b64b134677add316bba1a4f70457926154341f6b7ee23577262c153020503c2e"
Sep 17 23:21:22.177650 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:22.177631    2568 scope.go:117] "RemoveContainer" containerID="b64b134677add316bba1a4f70457926154341f6b7ee23577262c153020503c2e"
Sep 17 23:21:22.177956 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:21:22.177928    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64b134677add316bba1a4f70457926154341f6b7ee23577262c153020503c2e\": container with ID starting with b64b134677add316bba1a4f70457926154341f6b7ee23577262c153020503c2e not found: ID does not exist" containerID="b64b134677add316bba1a4f70457926154341f6b7ee23577262c153020503c2e"
Sep 17 23:21:22.178024 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:22.177967    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64b134677add316bba1a4f70457926154341f6b7ee23577262c153020503c2e"} err="failed to get container status \"b64b134677add316bba1a4f70457926154341f6b7ee23577262c153020503c2e\": rpc error: code = NotFound desc = could not find container \"b64b134677add316bba1a4f70457926154341f6b7ee23577262c153020503c2e\": container with ID starting with b64b134677add316bba1a4f70457926154341f6b7ee23577262c153020503c2e not found: ID does not exist"
Sep 17 23:21:22.185689 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:22.185661    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"]
Sep 17 23:21:22.194589 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:22.191855    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-61e623-786968fc8d-hwcdl"]
Sep 17 23:21:22.601174 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:22.601135    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:23.260194 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:23.260163    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:23.990339 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:23.990302    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8cf168f-32a7-4c0a-be24-69df43568155" path="/var/lib/kubelet/pods/a8cf168f-32a7-4c0a-be24-69df43568155/volumes"
Sep 17 23:21:24.038750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:24.038715    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:24.812257 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:24.812222    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:25.484417 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:25.484383    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:26.323167 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:26.323134    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:27.000124 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:27.000090    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:28.626368 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:28.626334    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:29.189911 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.189862    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"]
Sep 17 23:21:29.190141 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.190097    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk" podUID="102402db-1d69-45d5-ae13-87c03aa62f89" containerName="maas-api" containerID="cri-o://9f35b3deaa63fdd2e7992dff6404aca53ac759b04220a46e2c29d47cb9414c09" gracePeriod=30
Sep 17 23:21:29.219476 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.219446    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:29.442574 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.442513    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:21:29.590182 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.590149    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102402db-1d69-45d5-ae13-87c03aa62f89-maas-api-tls\") pod \"102402db-1d69-45d5-ae13-87c03aa62f89\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") "
Sep 17 23:21:29.590387 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.590208    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/102402db-1d69-45d5-ae13-87c03aa62f89-openshift-service-ca\") pod \"102402db-1d69-45d5-ae13-87c03aa62f89\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") "
Sep 17 23:21:29.590387 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.590336    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8ztw\" (UniqueName: \"kubernetes.io/projected/102402db-1d69-45d5-ae13-87c03aa62f89-kube-api-access-m8ztw\") pod \"102402db-1d69-45d5-ae13-87c03aa62f89\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") "
Sep 17 23:21:29.590387 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.590371    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102402db-1d69-45d5-ae13-87c03aa62f89-metrics-certs\") pod \"102402db-1d69-45d5-ae13-87c03aa62f89\" (UID: \"102402db-1d69-45d5-ae13-87c03aa62f89\") "
Sep 17 23:21:29.590620 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.590592    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/102402db-1d69-45d5-ae13-87c03aa62f89-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "102402db-1d69-45d5-ae13-87c03aa62f89" (UID: "102402db-1d69-45d5-ae13-87c03aa62f89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:21:29.592441 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.592412    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102402db-1d69-45d5-ae13-87c03aa62f89-kube-api-access-m8ztw" (OuterVolumeSpecName: "kube-api-access-m8ztw") pod "102402db-1d69-45d5-ae13-87c03aa62f89" (UID: "102402db-1d69-45d5-ae13-87c03aa62f89"). InnerVolumeSpecName "kube-api-access-m8ztw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:21:29.592441 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.592423    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102402db-1d69-45d5-ae13-87c03aa62f89-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "102402db-1d69-45d5-ae13-87c03aa62f89" (UID: "102402db-1d69-45d5-ae13-87c03aa62f89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:21:29.592570 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.592441    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102402db-1d69-45d5-ae13-87c03aa62f89-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "102402db-1d69-45d5-ae13-87c03aa62f89" (UID: "102402db-1d69-45d5-ae13-87c03aa62f89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:21:29.692127 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.692088    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8ztw\" (UniqueName: \"kubernetes.io/projected/102402db-1d69-45d5-ae13-87c03aa62f89-kube-api-access-m8ztw\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:21:29.692127 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.692123    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/102402db-1d69-45d5-ae13-87c03aa62f89-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:21:29.692127 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.692133    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/102402db-1d69-45d5-ae13-87c03aa62f89-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:21:29.692619 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:29.692145    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/102402db-1d69-45d5-ae13-87c03aa62f89-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:21:30.213818 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:30.213775    2568 generic.go:356] "Generic (PLEG): container finished" podID="102402db-1d69-45d5-ae13-87c03aa62f89" containerID="9f35b3deaa63fdd2e7992dff6404aca53ac759b04220a46e2c29d47cb9414c09" exitCode=0
Sep 17 23:21:30.213818 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:30.213817    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk" event={"ID":"102402db-1d69-45d5-ae13-87c03aa62f89","Type":"ContainerDied","Data":"9f35b3deaa63fdd2e7992dff6404aca53ac759b04220a46e2c29d47cb9414c09"}
Sep 17 23:21:30.214105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:30.213899    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk" event={"ID":"102402db-1d69-45d5-ae13-87c03aa62f89","Type":"ContainerDied","Data":"0cd1b479b3dafa116ef2e331a5487aed3f2e11b9d83a58bf8e57e23ae408bf00"}
Sep 17 23:21:30.214105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:30.213928    2568 scope.go:117] "RemoveContainer" containerID="9f35b3deaa63fdd2e7992dff6404aca53ac759b04220a46e2c29d47cb9414c09"
Sep 17 23:21:30.214105 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:30.213990    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"
Sep 17 23:21:30.223572 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:30.223551    2568 scope.go:117] "RemoveContainer" containerID="9f35b3deaa63fdd2e7992dff6404aca53ac759b04220a46e2c29d47cb9414c09"
Sep 17 23:21:30.223943 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:21:30.223921    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f35b3deaa63fdd2e7992dff6404aca53ac759b04220a46e2c29d47cb9414c09\": container with ID starting with 9f35b3deaa63fdd2e7992dff6404aca53ac759b04220a46e2c29d47cb9414c09 not found: ID does not exist" containerID="9f35b3deaa63fdd2e7992dff6404aca53ac759b04220a46e2c29d47cb9414c09"
Sep 17 23:21:30.224021 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:30.223956    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f35b3deaa63fdd2e7992dff6404aca53ac759b04220a46e2c29d47cb9414c09"} err="failed to get container status \"9f35b3deaa63fdd2e7992dff6404aca53ac759b04220a46e2c29d47cb9414c09\": rpc error: code = NotFound desc = could not find container \"9f35b3deaa63fdd2e7992dff6404aca53ac759b04220a46e2c29d47cb9414c09\": container with ID starting with 9f35b3deaa63fdd2e7992dff6404aca53ac759b04220a46e2c29d47cb9414c09 not found: ID does not exist"
Sep 17 23:21:30.237365 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:30.237337    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"]
Sep 17 23:21:30.239361 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:30.239333    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-46ccf8cf-7d9687fb76-pmgxk"]
Sep 17 23:21:30.427415 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:30.427387    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:30.649399 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:30.649369    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:31.394306 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:31.394273    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:31.861202 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:31.861170    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:31.991010 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:31.990978    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="102402db-1d69-45d5-ae13-87c03aa62f89" path="/var/lib/kubelet/pods/102402db-1d69-45d5-ae13-87c03aa62f89/volumes"
Sep 17 23:21:32.230247 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:32.230155    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:32.999521 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:32.999463    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:37.484929 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:37.484898    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:37.547832 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:37.547796    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:37.630394 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:37.630364    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:37.736508 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:37.736428    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:37.830817 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:37.830782    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:38.189114 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:38.189084    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:39.510428 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:39.510375    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:55.432240 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:21:55.432204    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:04.112531 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:04.112496    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:04.213068 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:04.213033    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:04.366235 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:04.366152    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:04.805638 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:04.805603    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:05.105421 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:05.105337    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:05.553231 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:05.553201    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:07.962698 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:07.962663    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:08.172546 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.172497    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:08.214721 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.214629    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"]
Sep 17 23:22:08.215149 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.215128    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="102402db-1d69-45d5-ae13-87c03aa62f89" containerName="maas-api"
Sep 17 23:22:08.215248 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.215151    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="102402db-1d69-45d5-ae13-87c03aa62f89" containerName="maas-api"
Sep 17 23:22:08.215248 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.215169    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8cf168f-32a7-4c0a-be24-69df43568155" containerName="maas-api"
Sep 17 23:22:08.215248 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.215177    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8cf168f-32a7-4c0a-be24-69df43568155" containerName="maas-api"
Sep 17 23:22:08.215464 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.215299    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="102402db-1d69-45d5-ae13-87c03aa62f89" containerName="maas-api"
Sep 17 23:22:08.215464 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.215316    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8cf168f-32a7-4c0a-be24-69df43568155" containerName="maas-api"
Sep 17 23:22:08.218478 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.218455    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.220813 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.220793    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-825f30\"/\"premium-w4-825f30-kserve-self-signed-certs\""
Sep 17 23:22:08.229192 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.229165    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"]
Sep 17 23:22:08.274994 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.274957    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:08.370091 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.370053    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:08.373554 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.373533    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-dshm\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.373651 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.373589    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-tls-certs\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.373651 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.373617    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-model-cache\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.373726 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.373685    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-486vn\" (UniqueName: \"kubernetes.io/projected/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-kube-api-access-486vn\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.373769 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.373732    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-home\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.373769 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.373760    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-tmp-dir\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.474468 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.474371    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-home\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.474468 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.474417    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-tmp-dir\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.474707 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.474474    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-dshm\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.474707 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.474520    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-tls-certs\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.474707 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.474554    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-model-cache\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.474707 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.474614    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486vn\" (UniqueName: \"kubernetes.io/projected/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-kube-api-access-486vn\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.474965 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.474890    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-home\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.474965 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.474924    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-tmp-dir\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.475084 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.474991    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-model-cache\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.476859 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.476827    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-dshm\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.477092 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.477074    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-tls-certs\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.489074 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.489052    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-486vn\" (UniqueName: \"kubernetes.io/projected/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-kube-api-access-486vn\") pod \"premium-w4-825f30-kserve-87fcb455d-54t52\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") " pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.529147 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.529109    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:08.804088 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:08.804061    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"]
Sep 17 23:22:08.805724 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:22:08.805698    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d35d4f5_288e_4f35_b646_d38e0c03ebf0.slice/crio-a1d05c9aa9a71d800bbc99da8a6522f9f5353ceb87611db23bdd77097019c946 WatchSource:0}: Error finding container a1d05c9aa9a71d800bbc99da8a6522f9f5353ceb87611db23bdd77097019c946: Status 404 returned error can't find the container with id a1d05c9aa9a71d800bbc99da8a6522f9f5353ceb87611db23bdd77097019c946
Sep 17 23:22:09.050523 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:09.050489    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:09.418251 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:09.418213    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52" event={"ID":"6d35d4f5-288e-4f35-b646-d38e0c03ebf0","Type":"ContainerStarted","Data":"9faf8e02d890277582e13db10e4df0c8682ca49b8aa6a86332fdb99657afafd5"}
Sep 17 23:22:09.418469 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:09.418257    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52" event={"ID":"6d35d4f5-288e-4f35-b646-d38e0c03ebf0","Type":"ContainerStarted","Data":"a1d05c9aa9a71d800bbc99da8a6522f9f5353ceb87611db23bdd77097019c946"}
Sep 17 23:22:09.441224 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:09.441173    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52" podStartSLOduration=1.441154332 podStartE2EDuration="1.441154332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:09.439611141 +0000 UTC m=+927.663286903" watchObservedRunningTime="2026-09-17 23:22:09.441154332 +0000 UTC m=+927.664830080"
Sep 17 23:22:09.930211 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:09.930174    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:10.239756 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:10.239677    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:10.345137 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:10.345106    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:10.440685 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:10.440656    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:10.906375 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:10.906346    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:11.950845 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:11.950813    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:12.010009 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:12.009978    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:12.110671 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:12.110637    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:12.429457 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:12.429411    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:12.517454 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:12.517408    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:18.529705 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:18.529654    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:18.529705 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:18.529721    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:18.542468 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:18.542438    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:19.480331 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:19.480300    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:22:26.320308 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:26.320268    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:31.410118 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:31.410083    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:32.698614 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:32.698579    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:32.811084 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:32.811051    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:32.903090 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:32.903056    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:34.235295 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:34.235263    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:34.951404 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:34.951361    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:35.712481 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:35.712437    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:36.514965 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:36.514935    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:37.291537 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:37.291496    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:38.096652 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:38.096612    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:38.241171 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:38.241126    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:43.383325 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:43.383292    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:44.154775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:44.154744    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:49.799321 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:49.799289    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:49.904233 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:49.904195    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:49.999248 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:49.999208    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:50.059613 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:50.059533    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:50.742791 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:50.742756    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:50.832357 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:50.832328    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:50.933703 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:50.933668    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:51.030958 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:51.030932    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:52.365574 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:52.365536    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:57.999016 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:57.998940    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:58.098922 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:58.098888    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:58.657963 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:22:58.657932    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:04.373995 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:04.373965    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:04.445322 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:04.445293    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:04.594840 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:04.594809    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:04.894814 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:04.894759    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:05.599032 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:05.599001    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:10.323526 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:10.323482    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:10.989223 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:10.989183    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:11.112229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:11.112195    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:11.385177 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:11.385141    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:12.138139 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:12.138110    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:12.958632 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:12.958590    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:13.737836 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:13.737795    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:14.539326 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:14.539292    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:15.344515 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:15.344483    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:15.448843 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:15.448809    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:15.851205 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:15.851146    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:15.950442 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:15.950407    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:38.641233 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:38.641197    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:39.570884 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:39.570836    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:40.339932 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:40.339894    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:40.998407 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:40.998376    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:41.048357 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:41.048329    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:41.662772 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:41.662739    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:41.762190 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:41.762161    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:41.861502 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:41.861474    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:42.073465 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.073432    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:42.529577 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.529541    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:42.598481 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.598448    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"]
Sep 17 23:23:42.601989 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.601966    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.606013 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.605992    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-35df6d\"/\"kube-root-ca.crt\""
Sep 17 23:23:42.606166 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.606045    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-35df6d\"/\"default-dockercfg-qk9xx\""
Sep 17 23:23:42.606924 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.606900    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-35df6d\"/\"explicit-model-35df6d-kserve-self-signed-certs\""
Sep 17 23:23:42.607045 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.606937    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-35df6d\"/\"openshift-service-ca.crt\""
Sep 17 23:23:42.628715 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.628684    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"]
Sep 17 23:23:42.675359 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.675314    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:42.751341 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.751308    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-dshm\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.751341 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.751345    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0774728b-fc7e-4b3a-92bb-173d85f06d5e-tls-certs\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.751578 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.751365    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls9j7\" (UniqueName: \"kubernetes.io/projected/0774728b-fc7e-4b3a-92bb-173d85f06d5e-kube-api-access-ls9j7\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.751578 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.751501    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-tmp-dir\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.751648 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.751583    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-home\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.751648 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.751623    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-model-cache\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.852734 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.852639    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-home\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.852734 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.852693    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-model-cache\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.853028 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.852746    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-dshm\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.853028 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.852764    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0774728b-fc7e-4b3a-92bb-173d85f06d5e-tls-certs\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.853028 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.852782    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls9j7\" (UniqueName: \"kubernetes.io/projected/0774728b-fc7e-4b3a-92bb-173d85f06d5e-kube-api-access-ls9j7\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.853028 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.852844    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-tmp-dir\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.853255 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.853164    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-home\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.853255 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.853214    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-model-cache\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.853327 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.853306    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-tmp-dir\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.855131 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.855112    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-dshm\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.855303 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.855284    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0774728b-fc7e-4b3a-92bb-173d85f06d5e-tls-certs\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.860994 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.860972    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls9j7\" (UniqueName: \"kubernetes.io/projected/0774728b-fc7e-4b3a-92bb-173d85f06d5e-kube-api-access-ls9j7\") pod \"explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") " pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:42.912107 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:42.912073    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:43.185894 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:43.185841    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"]
Sep 17 23:23:43.188578 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:23:43.188540    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0774728b_fc7e_4b3a_92bb_173d85f06d5e.slice/crio-467bf0911d017942453a414d8e9521bd6c9cb03931fac371a3e817b0dd821a08 WatchSource:0}: Error finding container 467bf0911d017942453a414d8e9521bd6c9cb03931fac371a3e817b0dd821a08: Status 404 returned error can't find the container with id 467bf0911d017942453a414d8e9521bd6c9cb03931fac371a3e817b0dd821a08
Sep 17 23:23:43.503146 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:43.503065    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:43.897270 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:43.897235    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg" event={"ID":"0774728b-fc7e-4b3a-92bb-173d85f06d5e","Type":"ContainerStarted","Data":"c2279095e53e837ce3741be0312f4d160c6404ab7f257139f87c713e338748e9"}
Sep 17 23:23:43.897652 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:43.897278    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg" event={"ID":"0774728b-fc7e-4b3a-92bb-173d85f06d5e","Type":"ContainerStarted","Data":"467bf0911d017942453a414d8e9521bd6c9cb03931fac371a3e817b0dd821a08"}
Sep 17 23:23:43.915623 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:43.915570    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg" podStartSLOduration=1.915555374 podStartE2EDuration="1.915555374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:43.913586029 +0000 UTC m=+1022.137261778" watchObservedRunningTime="2026-09-17 23:23:43.915555374 +0000 UTC m=+1022.139231123"
Sep 17 23:23:47.930763 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:47.930715    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:48.031174 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:48.031143    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:49.182838 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:49.182807    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:50.175035 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:50.174997    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:50.977823 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:50.977785    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:51.826402 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:51.826373    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:52.588794 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:52.588758    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:52.913100 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:52.913012    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:52.913100 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:52.913058    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:52.925822 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:52.925786    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:52.957993 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:52.957962    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:23:53.465152 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:53.465118    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:54.249010 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:54.248976    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:58.761530 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:58.761501    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:58.858286 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:58.858258    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:58.947614 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:58.947578    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:59.007502 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:59.007470    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:59.314259 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:59.314223    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:59.415293 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:23:59.415257    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:00.241507 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:00.241479    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:00.341589 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:00.341558    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:01.032687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:01.032658    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:01.792731 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:01.792700    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:02.553674 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:02.553643    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:03.219857 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:03.219824    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:04.970787 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:04.970753    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:05.246152 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:05.246056    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:05.346258 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:05.346226    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:05.519759 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:05.519651    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:05.830519 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:05.830484    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:05.931447 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:05.931403    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:06.050960 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:06.050919    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:06.565855 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:06.565822    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:11.575862 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:11.575827    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:11.632440 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:11.632407    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:11.714660 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:11.714621    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:11.824397 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:11.824365    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:11.953165 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:11.953076    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:14.330559 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:14.330515    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:16.261583 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:16.261551    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:16.876913 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:16.876846    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"]
Sep 17 23:24:16.903348 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:16.903321    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:17.255299 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:17.255215    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:17.892759 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:17.892722    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:17.994268 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:17.994236    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:18.732090 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:18.732046    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:18.833068 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:18.833018    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:19.075887 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:19.075843    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:19.298746 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:19.298709    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:19.625155 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:19.625119    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:20.474002 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:20.473959    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:21.383290 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:21.383252    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:21.906221 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:21.906183    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:21.999806 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:21.999770    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:22.278913 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:22.278823    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:23.120524 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:23.120483    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:23.918648 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:23.918599    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:24.631147 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:24.631115    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:25.402451 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:25.402354    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:26.025883 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.025813    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:26.095101 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.095072    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:26.195454 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.195421    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:26.266670 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.266630    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"]
Sep 17 23:24:26.270310 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.270282    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:26.270594 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.270578    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:26.273488 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.273464    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-b52922-serving-cert\""
Sep 17 23:24:26.280397 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.280376    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"]
Sep 17 23:24:26.317249 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.317218    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:26.317249 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.317258    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:26.317458 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.317432    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wlfx\" (UniqueName: \"kubernetes.io/projected/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-kube-api-access-6wlfx\") pod \"maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:26.317501 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.317486    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:26.339295 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.339262    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:26.407632 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.407596    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:26.418966 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.418928    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wlfx\" (UniqueName: \"kubernetes.io/projected/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-kube-api-access-6wlfx\") pod \"maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:26.419148 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.418985    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:26.419148 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.419005    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:26.419263 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.419146    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:26.419356 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:24:26.419335    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-serving-cert: secret "maas-api-e2e-shared-a-w2-b52922-serving-cert" not found
Sep 17 23:24:26.419592 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:24:26.419578    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-maas-api-tls podName:9c2012bd-9df1-48e9-9c40-17e7e4f43eb0 nodeName:}" failed. No retries permitted until 2026-09-17 23:24:26.919556709 +0000 UTC m=+1065.143232434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-maas-api-tls") pod "maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl" (UID: "9c2012bd-9df1-48e9-9c40-17e7e4f43eb0") : secret "maas-api-e2e-shared-a-w2-b52922-serving-cert" not found
Sep 17 23:24:26.419853 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.419825    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:26.421939 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.421915    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:26.427993 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.427971    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wlfx\" (UniqueName: \"kubernetes.io/projected/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-kube-api-access-6wlfx\") pod \"maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:26.478227 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.478191    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:26.645372 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.645284    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:26.914220 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.914129    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:26.924573 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.924539    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:26.927407 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:26.927335    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:27.022615 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:27.022580    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:27.182641 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:27.182546    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:27.381267 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:27.381233    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:27.467909 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:27.467884    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"]
Sep 17 23:24:27.470401 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:24:27.470358    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2012bd_9df1_48e9_9c40_17e7e4f43eb0.slice/crio-75f6da4fe602ce766e5d9b986d2ab0ee37dccf6d040cbcfadcb9b08606d94a8b WatchSource:0}: Error finding container 75f6da4fe602ce766e5d9b986d2ab0ee37dccf6d040cbcfadcb9b08606d94a8b: Status 404 returned error can't find the container with id 75f6da4fe602ce766e5d9b986d2ab0ee37dccf6d040cbcfadcb9b08606d94a8b
Sep 17 23:24:27.471886 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:27.471841    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:24:28.127855 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:28.127815    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl" event={"ID":"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0","Type":"ContainerStarted","Data":"67964f92433f62fb7fd3aa8f3c868ab12bfa3ea84358e8ac629f759fb007eb82"}
Sep 17 23:24:28.128051 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:28.127898    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl" event={"ID":"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0","Type":"ContainerStarted","Data":"75f6da4fe602ce766e5d9b986d2ab0ee37dccf6d040cbcfadcb9b08606d94a8b"}
Sep 17 23:24:28.128051 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:28.127938    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:28.150110 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:28.150059    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl" podStartSLOduration=1.69697666 podStartE2EDuration="2.150039395s" totalImagesPullingTime="453.062735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:24:28.148137265 +0000 UTC m=+1066.371813014" watchObservedRunningTime="2026-09-17 23:24:28.150039395 +0000 UTC m=+1066.373715142"
Sep 17 23:24:28.247905 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:28.247814    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:29.056666 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:29.056630    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:29.796510 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:29.796455    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:30.644808 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:30.644774    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:31.333795 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:31.333759    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:31.806656 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:31.806630    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:31.895913 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:31.895862    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg" podUID="0774728b-fc7e-4b3a-92bb-173d85f06d5e" containerName="main" containerID="cri-o://c2279095e53e837ce3741be0312f4d160c6404ab7f257139f87c713e338748e9" gracePeriod=45
Sep 17 23:24:32.050191 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.050158    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:32.141374 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.141349    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:24:32.149115 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.149083    2568 generic.go:356] "Generic (PLEG): container finished" podID="0774728b-fc7e-4b3a-92bb-173d85f06d5e" containerID="c2279095e53e837ce3741be0312f4d160c6404ab7f257139f87c713e338748e9" exitCode=0
Sep 17 23:24:32.149266 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.149122    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"
Sep 17 23:24:32.149266 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.149156    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg" event={"ID":"0774728b-fc7e-4b3a-92bb-173d85f06d5e","Type":"ContainerDied","Data":"c2279095e53e837ce3741be0312f4d160c6404ab7f257139f87c713e338748e9"}
Sep 17 23:24:32.149266 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.149204    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg" event={"ID":"0774728b-fc7e-4b3a-92bb-173d85f06d5e","Type":"ContainerDied","Data":"467bf0911d017942453a414d8e9521bd6c9cb03931fac371a3e817b0dd821a08"}
Sep 17 23:24:32.149266 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.149223    2568 scope.go:117] "RemoveContainer" containerID="c2279095e53e837ce3741be0312f4d160c6404ab7f257139f87c713e338748e9"
Sep 17 23:24:32.157683 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.157663    2568 scope.go:117] "RemoveContainer" containerID="c2279095e53e837ce3741be0312f4d160c6404ab7f257139f87c713e338748e9"
Sep 17 23:24:32.157952 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:24:32.157931    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2279095e53e837ce3741be0312f4d160c6404ab7f257139f87c713e338748e9\": container with ID starting with c2279095e53e837ce3741be0312f4d160c6404ab7f257139f87c713e338748e9 not found: ID does not exist" containerID="c2279095e53e837ce3741be0312f4d160c6404ab7f257139f87c713e338748e9"
Sep 17 23:24:32.158013 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.157962    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2279095e53e837ce3741be0312f4d160c6404ab7f257139f87c713e338748e9"} err="failed to get container status \"c2279095e53e837ce3741be0312f4d160c6404ab7f257139f87c713e338748e9\": rpc error: code = NotFound desc = could not find container \"c2279095e53e837ce3741be0312f4d160c6404ab7f257139f87c713e338748e9\": container with ID starting with c2279095e53e837ce3741be0312f4d160c6404ab7f257139f87c713e338748e9 not found: ID does not exist"
Sep 17 23:24:32.180986 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.180954    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-model-cache\") pod \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") "
Sep 17 23:24:32.181163 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.181011    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-home\") pod \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") "
Sep 17 23:24:32.181163 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.181033    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-tmp-dir\") pod \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") "
Sep 17 23:24:32.181163 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.181063    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-dshm\") pod \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") "
Sep 17 23:24:32.181163 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.181086    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0774728b-fc7e-4b3a-92bb-173d85f06d5e-tls-certs\") pod \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") "
Sep 17 23:24:32.181163 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.181116    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls9j7\" (UniqueName: \"kubernetes.io/projected/0774728b-fc7e-4b3a-92bb-173d85f06d5e-kube-api-access-ls9j7\") pod \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\" (UID: \"0774728b-fc7e-4b3a-92bb-173d85f06d5e\") "
Sep 17 23:24:32.181432 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.181290    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "0774728b-fc7e-4b3a-92bb-173d85f06d5e" (UID: "0774728b-fc7e-4b3a-92bb-173d85f06d5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:24:32.181432 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.181346    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0774728b-fc7e-4b3a-92bb-173d85f06d5e" (UID: "0774728b-fc7e-4b3a-92bb-173d85f06d5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:24:32.181432 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.181399    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-home" (OuterVolumeSpecName: "home") pod "0774728b-fc7e-4b3a-92bb-173d85f06d5e" (UID: "0774728b-fc7e-4b3a-92bb-173d85f06d5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:24:32.181585 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.181519    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-model-cache\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:24:32.181585 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.181539    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-home\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:24:32.181585 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.181552    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-tmp-dir\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:24:32.183313 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.183292    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-dshm" (OuterVolumeSpecName: "dshm") pod "0774728b-fc7e-4b3a-92bb-173d85f06d5e" (UID: "0774728b-fc7e-4b3a-92bb-173d85f06d5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:24:32.183430 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.183385    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0774728b-fc7e-4b3a-92bb-173d85f06d5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0774728b-fc7e-4b3a-92bb-173d85f06d5e" (UID: "0774728b-fc7e-4b3a-92bb-173d85f06d5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:24:32.183697 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.183672    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0774728b-fc7e-4b3a-92bb-173d85f06d5e-kube-api-access-ls9j7" (OuterVolumeSpecName: "kube-api-access-ls9j7") pod "0774728b-fc7e-4b3a-92bb-173d85f06d5e" (UID: "0774728b-fc7e-4b3a-92bb-173d85f06d5e"). InnerVolumeSpecName "kube-api-access-ls9j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:24:32.282312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.282272    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0774728b-fc7e-4b3a-92bb-173d85f06d5e-dshm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:24:32.282312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.282305    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0774728b-fc7e-4b3a-92bb-173d85f06d5e-tls-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:24:32.282312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.282317    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls9j7\" (UniqueName: \"kubernetes.io/projected/0774728b-fc7e-4b3a-92bb-173d85f06d5e-kube-api-access-ls9j7\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:24:32.472214 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.472179    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"]
Sep 17 23:24:32.477074 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:32.477048    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-35df6d/explicit-model-35df6d-kserve-6dc9f6bd5c-mmxkg"]
Sep 17 23:24:33.990413 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:33.990374    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0774728b-fc7e-4b3a-92bb-173d85f06d5e" path="/var/lib/kubelet/pods/0774728b-fc7e-4b3a-92bb-173d85f06d5e/volumes"
Sep 17 23:24:34.138050 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:34.138019    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:24:37.864935 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:37.864894    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:38.112304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:38.112274    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:38.566939 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:38.566859    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:38.860209 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:38.860135    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:39.687365 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:39.687328    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:40.421025 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:40.420988    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:40.520630 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:40.520598    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:40.622008 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:40.621982    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:47.332212 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:47.332181    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:48.874272 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:48.874222    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:49.739817 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:49.739785    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:50.451242 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:50.451202    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:51.236557 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:51.236526    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:51.914363 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:51.914331    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:52.021739 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:52.021710    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:52.246281 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:52.246202    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:52.539343 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:52.539313    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:52.674947 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:52.674904    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:52.815920 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:52.815777    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:56.048859 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:56.048828    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:56.159691 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:56.159664    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:56.280755 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:56.280723    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:57.212986 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:57.212956    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:58.232452 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:58.232420    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:58.945339 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:58.945272    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:59.732890 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:24:59.732842    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:00.462639 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:00.462605    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:00.551320 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:00.551263    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:00.759333 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:00.759229    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:00.951271 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:00.951228    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:01.050014 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:01.049981    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:01.135649 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:01.135618    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:23.526016 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:23.525958    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:24.169089 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:24.169060    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:24.822013 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:24.821979    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:25.550324 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:25.550289    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:26.176310 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:26.176280    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:26.379532 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:26.379493    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:27.489919 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:27.489875    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:28.154634 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:28.154599    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:28.914486 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:28.914453    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:29.610503 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:29.610468    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:30.317213 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:30.317178    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:31.273625 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:31.273593    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:35.638886 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:35.638836    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:35.741901 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:35.741855    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:35.982252 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:35.982173    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:36.223812 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.223777    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:36.273388 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.273355    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"]
Sep 17 23:25:36.273638 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.273613    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk" podUID="63c8522c-fc35-48b2-b1a6-6fb35ca77784" containerName="maas-api" containerID="cri-o://eb4919f1122557de259fa7fdbc07fa0027aa6eb174f9fe4e4d148a7c32da111e" gracePeriod=30
Sep 17 23:25:36.300009 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.299975    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:36.367638 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.367604    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:36.471908 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.471854    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:36.492849 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.492819    2568 generic.go:356] "Generic (PLEG): container finished" podID="63c8522c-fc35-48b2-b1a6-6fb35ca77784" containerID="eb4919f1122557de259fa7fdbc07fa0027aa6eb174f9fe4e4d148a7c32da111e" exitCode=0
Sep 17 23:25:36.493021 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.492905    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk" event={"ID":"63c8522c-fc35-48b2-b1a6-6fb35ca77784","Type":"ContainerDied","Data":"eb4919f1122557de259fa7fdbc07fa0027aa6eb174f9fe4e4d148a7c32da111e"}
Sep 17 23:25:36.525551 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.525489    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:25:36.548555 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.548521    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:36.635047 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.635015    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-maas-api-tls\") pod \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") "
Sep 17 23:25:36.635239 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.635109    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j646q\" (UniqueName: \"kubernetes.io/projected/63c8522c-fc35-48b2-b1a6-6fb35ca77784-kube-api-access-j646q\") pod \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") "
Sep 17 23:25:36.635239 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.635167    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-metrics-certs\") pod \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") "
Sep 17 23:25:36.635239 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.635220    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c8522c-fc35-48b2-b1a6-6fb35ca77784-openshift-service-ca\") pod \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\" (UID: \"63c8522c-fc35-48b2-b1a6-6fb35ca77784\") "
Sep 17 23:25:36.635625 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.635594    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c8522c-fc35-48b2-b1a6-6fb35ca77784-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63c8522c-fc35-48b2-b1a6-6fb35ca77784" (UID: "63c8522c-fc35-48b2-b1a6-6fb35ca77784"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:25:36.637376 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.637345    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63c8522c-fc35-48b2-b1a6-6fb35ca77784" (UID: "63c8522c-fc35-48b2-b1a6-6fb35ca77784"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:25:36.637963 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.637942    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "63c8522c-fc35-48b2-b1a6-6fb35ca77784" (UID: "63c8522c-fc35-48b2-b1a6-6fb35ca77784"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:25:36.638061 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.638039    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c8522c-fc35-48b2-b1a6-6fb35ca77784-kube-api-access-j646q" (OuterVolumeSpecName: "kube-api-access-j646q") pod "63c8522c-fc35-48b2-b1a6-6fb35ca77784" (UID: "63c8522c-fc35-48b2-b1a6-6fb35ca77784"). InnerVolumeSpecName "kube-api-access-j646q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:25:36.736471 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.736422    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:25:36.736471 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.736460    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c8522c-fc35-48b2-b1a6-6fb35ca77784-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:25:36.736471 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.736470    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c8522c-fc35-48b2-b1a6-6fb35ca77784-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:25:36.736981 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.736483    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j646q\" (UniqueName: \"kubernetes.io/projected/63c8522c-fc35-48b2-b1a6-6fb35ca77784-kube-api-access-j646q\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:25:36.856110 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.856021    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:36.957455 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:36.957423    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:37.500298 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:37.500265    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"
Sep 17 23:25:37.500507 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:37.500314    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk" event={"ID":"63c8522c-fc35-48b2-b1a6-6fb35ca77784","Type":"ContainerDied","Data":"ddc10bd5244cde40c3ace7cb8400a35972a326731271eeabe4e295d1011446d0"}
Sep 17 23:25:37.500507 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:37.500387    2568 scope.go:117] "RemoveContainer" containerID="eb4919f1122557de259fa7fdbc07fa0027aa6eb174f9fe4e4d148a7c32da111e"
Sep 17 23:25:37.522359 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:37.522322    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"]
Sep 17 23:25:37.529037 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:37.529010    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-48d98f-55b6584df4-vrdnk"]
Sep 17 23:25:37.807663 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:37.807631    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:37.990839 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:37.990805    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63c8522c-fc35-48b2-b1a6-6fb35ca77784" path="/var/lib/kubelet/pods/63c8522c-fc35-48b2-b1a6-6fb35ca77784/volumes"
Sep 17 23:25:38.553356 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:38.553315    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:39.342802 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:39.342769    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:39.470743 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:39.470716    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:39.590930 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:39.590895    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:40.238145 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:40.238109    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:41.037638 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.037609    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:41.821212 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.821181    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"]
Sep 17 23:25:41.821598 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.821585    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63c8522c-fc35-48b2-b1a6-6fb35ca77784" containerName="maas-api"
Sep 17 23:25:41.821643 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.821600    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="63c8522c-fc35-48b2-b1a6-6fb35ca77784" containerName="maas-api"
Sep 17 23:25:41.821643 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.821622    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0774728b-fc7e-4b3a-92bb-173d85f06d5e" containerName="main"
Sep 17 23:25:41.821643 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.821628    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0774728b-fc7e-4b3a-92bb-173d85f06d5e" containerName="main"
Sep 17 23:25:41.821737 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.821704    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0774728b-fc7e-4b3a-92bb-173d85f06d5e" containerName="main"
Sep 17 23:25:41.821737 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.821716    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="63c8522c-fc35-48b2-b1a6-6fb35ca77784" containerName="maas-api"
Sep 17 23:25:41.826163 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.826144    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:41.829231 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.829209    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-09450b8d-serving-cert\""
Sep 17 23:25:41.858457 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.858424    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"]
Sep 17 23:25:41.889580 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.889547    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa4414e-4f92-40af-9f52-456fd8d4984a-maas-api-tls\") pod \"maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:41.889765 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.889605    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7rnm\" (UniqueName: \"kubernetes.io/projected/3fa4414e-4f92-40af-9f52-456fd8d4984a-kube-api-access-m7rnm\") pod \"maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:41.889765 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.889710    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4414e-4f92-40af-9f52-456fd8d4984a-metrics-certs\") pod \"maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:41.889765 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.889738    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa4414e-4f92-40af-9f52-456fd8d4984a-openshift-service-ca\") pod \"maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:41.896756 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.896725    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:41.897591 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.897567    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"]
Sep 17 23:25:41.983946 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.983911    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:41.990303 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.990276    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa4414e-4f92-40af-9f52-456fd8d4984a-maas-api-tls\") pod \"maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:41.990463 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.990330    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rnm\" (UniqueName: \"kubernetes.io/projected/3fa4414e-4f92-40af-9f52-456fd8d4984a-kube-api-access-m7rnm\") pod \"maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:41.990463 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.990404    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4414e-4f92-40af-9f52-456fd8d4984a-metrics-certs\") pod \"maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:41.990463 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.990438    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa4414e-4f92-40af-9f52-456fd8d4984a-openshift-service-ca\") pod \"maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:41.991149 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.991122    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa4414e-4f92-40af-9f52-456fd8d4984a-openshift-service-ca\") pod \"maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:41.992854 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.992833    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4414e-4f92-40af-9f52-456fd8d4984a-metrics-certs\") pod \"maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:41.992975 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.992900    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa4414e-4f92-40af-9f52-456fd8d4984a-maas-api-tls\") pod \"maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:41.998836 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:41.998810    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7rnm\" (UniqueName: \"kubernetes.io/projected/3fa4414e-4f92-40af-9f52-456fd8d4984a-kube-api-access-m7rnm\") pod \"maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:42.084119 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:42.084024    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:42.136740 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:42.136703    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:42.378548 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:42.378454    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:42.417728 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:42.417702    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"]
Sep 17 23:25:42.419841 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:25:42.419816    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa4414e_4f92_40af_9f52_456fd8d4984a.slice/crio-d4b141446625529af2afc1c75bf936398e95f16e7fc4fca4b4aa93dbe9d7e510 WatchSource:0}: Error finding container d4b141446625529af2afc1c75bf936398e95f16e7fc4fca4b4aa93dbe9d7e510: Status 404 returned error can't find the container with id d4b141446625529af2afc1c75bf936398e95f16e7fc4fca4b4aa93dbe9d7e510
Sep 17 23:25:42.477506 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:42.477473    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:42.528777 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:42.528743    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6" event={"ID":"3fa4414e-4f92-40af-9f52-456fd8d4984a","Type":"ContainerStarted","Data":"d4b141446625529af2afc1c75bf936398e95f16e7fc4fca4b4aa93dbe9d7e510"}
Sep 17 23:25:42.920135 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:42.920101    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:43.536450 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:43.536398    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6" event={"ID":"3fa4414e-4f92-40af-9f52-456fd8d4984a","Type":"ContainerStarted","Data":"458b26bfbeb68907e952d1af35e77b2c78cb3db0458883829854d0c37d9e38d6"}
Sep 17 23:25:43.537144 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:43.537116    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:43.557834 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:43.557764    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6" podStartSLOduration=2.070563915 podStartE2EDuration="2.557740208s" totalImagesPullingTime="487.176293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:43.556088252 +0000 UTC m=+1141.779764010" watchObservedRunningTime="2026-09-17 23:25:43.557740208 +0000 UTC m=+1141.781415957"
Sep 17 23:25:43.728408 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:43.728318    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:44.525204 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:44.525161    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:45.244124 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:45.244082    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:49.547616 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:49.547579    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:25:54.423017 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:54.422982    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:54.485655 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:54.485621    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:56.917491 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:56.917400    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld" podUID="667a1ddb-2e75-45ae-8f61-9ab806b2b6fc" containerName="main" containerID="cri-o://2e7c744dad0586ee2706e7facfdcdea2139d9add194b4a5c2ee824e9599c2d8c" gracePeriod=45
Sep 17 23:25:57.169231 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.169161    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:25:57.251730 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.251689    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-model-cache\") pod \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") "
Sep 17 23:25:57.251914 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.251740    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-dshm\") pod \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") "
Sep 17 23:25:57.251914 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.251764    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-tmp-dir\") pod \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") "
Sep 17 23:25:57.251914 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.251804    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sd97\" (UniqueName: \"kubernetes.io/projected/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-kube-api-access-5sd97\") pod \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") "
Sep 17 23:25:57.251914 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.251890    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-home\") pod \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") "
Sep 17 23:25:57.252058 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.251977    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "667a1ddb-2e75-45ae-8f61-9ab806b2b6fc" (UID: "667a1ddb-2e75-45ae-8f61-9ab806b2b6fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:57.252058 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.251997    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-tls-certs\") pod \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\" (UID: \"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc\") "
Sep 17 23:25:57.252141 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.252063    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "667a1ddb-2e75-45ae-8f61-9ab806b2b6fc" (UID: "667a1ddb-2e75-45ae-8f61-9ab806b2b6fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:57.252183 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.252135    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-home" (OuterVolumeSpecName: "home") pod "667a1ddb-2e75-45ae-8f61-9ab806b2b6fc" (UID: "667a1ddb-2e75-45ae-8f61-9ab806b2b6fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:57.252342 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.252328    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-model-cache\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:25:57.252342 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.252343    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-tmp-dir\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:25:57.252532 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.252351    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-home\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:25:57.254008 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.253981    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-kube-api-access-5sd97" (OuterVolumeSpecName: "kube-api-access-5sd97") pod "667a1ddb-2e75-45ae-8f61-9ab806b2b6fc" (UID: "667a1ddb-2e75-45ae-8f61-9ab806b2b6fc"). InnerVolumeSpecName "kube-api-access-5sd97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:25:57.254139 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.254010    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-dshm" (OuterVolumeSpecName: "dshm") pod "667a1ddb-2e75-45ae-8f61-9ab806b2b6fc" (UID: "667a1ddb-2e75-45ae-8f61-9ab806b2b6fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:57.254139 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.254016    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "667a1ddb-2e75-45ae-8f61-9ab806b2b6fc" (UID: "667a1ddb-2e75-45ae-8f61-9ab806b2b6fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:25:57.353420 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.353389    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-dshm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:25:57.353420 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.353419    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sd97\" (UniqueName: \"kubernetes.io/projected/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-kube-api-access-5sd97\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:25:57.353420 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.353434    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc-tls-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:25:57.618110 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.618073    2568 generic.go:356] "Generic (PLEG): container finished" podID="667a1ddb-2e75-45ae-8f61-9ab806b2b6fc" containerID="2e7c744dad0586ee2706e7facfdcdea2139d9add194b4a5c2ee824e9599c2d8c" exitCode=0
Sep 17 23:25:57.618304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.618117    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"
Sep 17 23:25:57.618304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.618165    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld" event={"ID":"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc","Type":"ContainerDied","Data":"2e7c744dad0586ee2706e7facfdcdea2139d9add194b4a5c2ee824e9599c2d8c"}
Sep 17 23:25:57.618304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.618203    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld" event={"ID":"667a1ddb-2e75-45ae-8f61-9ab806b2b6fc","Type":"ContainerDied","Data":"7490fb11c225f2a27212af43b515a3368f9a87baf9ab659681657793e8a15757"}
Sep 17 23:25:57.618304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.618223    2568 scope.go:117] "RemoveContainer" containerID="2e7c744dad0586ee2706e7facfdcdea2139d9add194b4a5c2ee824e9599c2d8c"
Sep 17 23:25:57.627281 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.627265    2568 scope.go:117] "RemoveContainer" containerID="2e7c744dad0586ee2706e7facfdcdea2139d9add194b4a5c2ee824e9599c2d8c"
Sep 17 23:25:57.627562 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:25:57.627545    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7c744dad0586ee2706e7facfdcdea2139d9add194b4a5c2ee824e9599c2d8c\": container with ID starting with 2e7c744dad0586ee2706e7facfdcdea2139d9add194b4a5c2ee824e9599c2d8c not found: ID does not exist" containerID="2e7c744dad0586ee2706e7facfdcdea2139d9add194b4a5c2ee824e9599c2d8c"
Sep 17 23:25:57.627620 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.627571    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7c744dad0586ee2706e7facfdcdea2139d9add194b4a5c2ee824e9599c2d8c"} err="failed to get container status \"2e7c744dad0586ee2706e7facfdcdea2139d9add194b4a5c2ee824e9599c2d8c\": rpc error: code = NotFound desc = could not find container \"2e7c744dad0586ee2706e7facfdcdea2139d9add194b4a5c2ee824e9599c2d8c\": container with ID starting with 2e7c744dad0586ee2706e7facfdcdea2139d9add194b4a5c2ee824e9599c2d8c not found: ID does not exist"
Sep 17 23:25:57.639662 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.639631    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"]
Sep 17 23:25:57.641568 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.641550    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-48d98f/test-model-48d98f-kserve-b6dc85d99-bz7ld"]
Sep 17 23:25:57.990650 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:25:57.990570    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="667a1ddb-2e75-45ae-8f61-9ab806b2b6fc" path="/var/lib/kubelet/pods/667a1ddb-2e75-45ae-8f61-9ab806b2b6fc/volumes"
Sep 17 23:26:00.396710 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:00.396678    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:03.954522 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:03.954490    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:04.418053 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:04.418021    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:04.687032 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:04.686952    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:05.670942 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:05.670912    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:06.345085 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:06.345050    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:07.141429 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:07.141397    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:07.828169 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:07.828107    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:08.493528 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:08.493487    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:09.423213 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:09.423183    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:09.524115 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:09.524082    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:18.773381 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:18.773346    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:19.118304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:19.118215    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:19.806991 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:19.806957    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:20.639092 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:20.639059    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:21.306087 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.306052    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"]
Sep 17 23:26:21.306491 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.306471    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667a1ddb-2e75-45ae-8f61-9ab806b2b6fc" containerName="main"
Sep 17 23:26:21.306491 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.306485    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="667a1ddb-2e75-45ae-8f61-9ab806b2b6fc" containerName="main"
Sep 17 23:26:21.306607 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.306565    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="667a1ddb-2e75-45ae-8f61-9ab806b2b6fc" containerName="main"
Sep 17 23:26:21.310014 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.309992    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.313672 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.313636    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-be3eb4\"/\"openshift-service-ca.crt\""
Sep 17 23:26:21.313811 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.313682    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-be3eb4\"/\"noten-model-be3eb4-kserve-self-signed-certs\""
Sep 17 23:26:21.313811 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.313698    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-be3eb4\"/\"default-dockercfg-csl2d\""
Sep 17 23:26:21.313811 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.313648    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-be3eb4\"/\"kube-root-ca.crt\""
Sep 17 23:26:21.320246 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.320225    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"]
Sep 17 23:26:21.343445 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.343409    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:21.390483 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.390459    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-dshm\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.390609 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.390519    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-model-cache\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.390609 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.390541    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmnb\" (UniqueName: \"kubernetes.io/projected/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-kube-api-access-ncmnb\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.390609 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.390575    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-tls-certs\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.390609 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.390595    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-home\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.390776 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.390633    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-tmp-dir\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.455966 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.455938    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:21.491311 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.491283    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-model-cache\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.491311 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.491311    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmnb\" (UniqueName: \"kubernetes.io/projected/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-kube-api-access-ncmnb\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.491497 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.491330    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-tls-certs\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.491497 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.491446    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-home\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.491617 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.491493    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-tmp-dir\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.491680 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.491656    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-dshm\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.491750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.491732    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-model-cache\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.491809 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.491792    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-home\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.491915 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.491893    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-tmp-dir\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.493757 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.493730    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-dshm\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.493900 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.493884    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-tls-certs\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.498750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.498732    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmnb\" (UniqueName: \"kubernetes.io/projected/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-kube-api-access-ncmnb\") pod \"noten-model-be3eb4-kserve-74bf6698dc-qdc9d\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") " pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.620746 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.620655    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:21.892753 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:21.892724    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"]
Sep 17 23:26:21.895192 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:26:21.895164    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb2b0d85_9e2a_44cc_9f6a_3d9d5cd79c1f.slice/crio-c060037faaee9a86307d3c3d4c51fa8eaa2d6461a8884b15d5b9ca516b2b7f34 WatchSource:0}: Error finding container c060037faaee9a86307d3c3d4c51fa8eaa2d6461a8884b15d5b9ca516b2b7f34: Status 404 returned error can't find the container with id c060037faaee9a86307d3c3d4c51fa8eaa2d6461a8884b15d5b9ca516b2b7f34
Sep 17 23:26:22.539355 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:22.539321    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:22.748226 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:22.748189    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d" event={"ID":"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f","Type":"ContainerStarted","Data":"789c401bb125776ce656d3e1c1a39345d629232817c667de67ca4c2f59f7376c"}
Sep 17 23:26:22.748390 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:22.748239    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d" event={"ID":"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f","Type":"ContainerStarted","Data":"c060037faaee9a86307d3c3d4c51fa8eaa2d6461a8884b15d5b9ca516b2b7f34"}
Sep 17 23:26:22.765269 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:22.765225    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d" podStartSLOduration=1.7652064140000001 podStartE2EDuration="1.765206414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:26:22.763846043 +0000 UTC m=+1180.987521791" watchObservedRunningTime="2026-09-17 23:26:22.765206414 +0000 UTC m=+1180.988882161"
Sep 17 23:26:23.242623 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:23.242592    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:23.940696 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:23.940667    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:24.212320 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:24.212243    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:24.695588 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:24.695557    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:25.325547 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:25.325517    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:30.019194 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.019162    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"]
Sep 17 23:26:30.019762 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.019402    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k" podUID="8387bbd7-2c95-4aea-94d7-2ab845e0e822" containerName="maas-api" containerID="cri-o://7a53b3dbb7b3e78932777d29eabc0c73d2aa091c3c06c98cfe14957dbbd75c43" gracePeriod=30
Sep 17 23:26:30.038005 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.037981    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:30.104512 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.104479    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:30.171885 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.171824    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:30.236658 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.236625    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:30.273365 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.273350    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:26:30.295126 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.295095    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:30.377503 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.377471    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl8dw\" (UniqueName: \"kubernetes.io/projected/8387bbd7-2c95-4aea-94d7-2ab845e0e822-kube-api-access-hl8dw\") pod \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") "
Sep 17 23:26:30.377684 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.377548    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8387bbd7-2c95-4aea-94d7-2ab845e0e822-metrics-certs\") pod \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") "
Sep 17 23:26:30.377684 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.377593    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8387bbd7-2c95-4aea-94d7-2ab845e0e822-openshift-service-ca\") pod \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") "
Sep 17 23:26:30.377684 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.377607    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8387bbd7-2c95-4aea-94d7-2ab845e0e822-maas-api-tls\") pod \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\" (UID: \"8387bbd7-2c95-4aea-94d7-2ab845e0e822\") "
Sep 17 23:26:30.377994 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.377956    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8387bbd7-2c95-4aea-94d7-2ab845e0e822-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8387bbd7-2c95-4aea-94d7-2ab845e0e822" (UID: "8387bbd7-2c95-4aea-94d7-2ab845e0e822"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:26:30.379724 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.379691    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8387bbd7-2c95-4aea-94d7-2ab845e0e822-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8387bbd7-2c95-4aea-94d7-2ab845e0e822" (UID: "8387bbd7-2c95-4aea-94d7-2ab845e0e822"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:26:30.379827 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.379737    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8387bbd7-2c95-4aea-94d7-2ab845e0e822-kube-api-access-hl8dw" (OuterVolumeSpecName: "kube-api-access-hl8dw") pod "8387bbd7-2c95-4aea-94d7-2ab845e0e822" (UID: "8387bbd7-2c95-4aea-94d7-2ab845e0e822"). InnerVolumeSpecName "kube-api-access-hl8dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:26:30.379827 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.379741    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8387bbd7-2c95-4aea-94d7-2ab845e0e822-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8387bbd7-2c95-4aea-94d7-2ab845e0e822" (UID: "8387bbd7-2c95-4aea-94d7-2ab845e0e822"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:26:30.478909 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.478849    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl8dw\" (UniqueName: \"kubernetes.io/projected/8387bbd7-2c95-4aea-94d7-2ab845e0e822-kube-api-access-hl8dw\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:26:30.478909 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.478904    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8387bbd7-2c95-4aea-94d7-2ab845e0e822-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:26:30.479100 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.478921    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8387bbd7-2c95-4aea-94d7-2ab845e0e822-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:26:30.479100 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.478933    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8387bbd7-2c95-4aea-94d7-2ab845e0e822-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:26:30.577651 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.577581    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:30.789423 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.789386    2568 generic.go:356] "Generic (PLEG): container finished" podID="8387bbd7-2c95-4aea-94d7-2ab845e0e822" containerID="7a53b3dbb7b3e78932777d29eabc0c73d2aa091c3c06c98cfe14957dbbd75c43" exitCode=0
Sep 17 23:26:30.789622 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.789442    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"
Sep 17 23:26:30.789622 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.789469    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k" event={"ID":"8387bbd7-2c95-4aea-94d7-2ab845e0e822","Type":"ContainerDied","Data":"7a53b3dbb7b3e78932777d29eabc0c73d2aa091c3c06c98cfe14957dbbd75c43"}
Sep 17 23:26:30.789622 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.789552    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k" event={"ID":"8387bbd7-2c95-4aea-94d7-2ab845e0e822","Type":"ContainerDied","Data":"5a29ee7ad250e71c97dd2ae6c56c6b1a0bb4a87096a66931336dbd5616c35dc0"}
Sep 17 23:26:30.789622 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.789583    2568 scope.go:117] "RemoveContainer" containerID="7a53b3dbb7b3e78932777d29eabc0c73d2aa091c3c06c98cfe14957dbbd75c43"
Sep 17 23:26:30.800162 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.800142    2568 scope.go:117] "RemoveContainer" containerID="7a53b3dbb7b3e78932777d29eabc0c73d2aa091c3c06c98cfe14957dbbd75c43"
Sep 17 23:26:30.800710 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:26:30.800681    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a53b3dbb7b3e78932777d29eabc0c73d2aa091c3c06c98cfe14957dbbd75c43\": container with ID starting with 7a53b3dbb7b3e78932777d29eabc0c73d2aa091c3c06c98cfe14957dbbd75c43 not found: ID does not exist" containerID="7a53b3dbb7b3e78932777d29eabc0c73d2aa091c3c06c98cfe14957dbbd75c43"
Sep 17 23:26:30.800802 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.800719    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a53b3dbb7b3e78932777d29eabc0c73d2aa091c3c06c98cfe14957dbbd75c43"} err="failed to get container status \"7a53b3dbb7b3e78932777d29eabc0c73d2aa091c3c06c98cfe14957dbbd75c43\": rpc error: code = NotFound desc = could not find container \"7a53b3dbb7b3e78932777d29eabc0c73d2aa091c3c06c98cfe14957dbbd75c43\": container with ID starting with 7a53b3dbb7b3e78932777d29eabc0c73d2aa091c3c06c98cfe14957dbbd75c43 not found: ID does not exist"
Sep 17 23:26:30.812612 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.812587    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"]
Sep 17 23:26:30.816884 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:30.816853    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f38e68-76676b75d6-qjz6k"]
Sep 17 23:26:31.621284 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:31.621250    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:31.621284 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:31.621292    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:31.634176 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:31.634148    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:31.806472 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:31.806441    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:26:31.990016 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:31.989924    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8387bbd7-2c95-4aea-94d7-2ab845e0e822" path="/var/lib/kubelet/pods/8387bbd7-2c95-4aea-94d7-2ab845e0e822/volumes"
Sep 17 23:26:32.063284 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:32.063248    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:32.220122 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:32.220076    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:32.862327 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:32.862287    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:33.495228 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:33.495198    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:34.236392 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:34.236364    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:35.650639 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:35.650609    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:37.695345 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.695316    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"]
Sep 17 23:26:37.695765 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.695752    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8387bbd7-2c95-4aea-94d7-2ab845e0e822" containerName="maas-api"
Sep 17 23:26:37.695765 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.695765    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8387bbd7-2c95-4aea-94d7-2ab845e0e822" containerName="maas-api"
Sep 17 23:26:37.695893 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.695829    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8387bbd7-2c95-4aea-94d7-2ab845e0e822" containerName="maas-api"
Sep 17 23:26:37.699031 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.699015    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.701498 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.701476    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-825f30\"/\"e2e-conflict-b-3df45c8d-kserve-self-signed-certs\""
Sep 17 23:26:37.712026 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.711997    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"]
Sep 17 23:26:37.735006 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.734971    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:37.855440 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.855411    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37473ad-4cfc-4df2-9215-693d80954676-tls-certs\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.855614 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.855448    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-tmp-dir\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.855614 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.855477    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-model-cache\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.855614 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.855577    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-dshm\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.855725 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.855615    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dh7x\" (UniqueName: \"kubernetes.io/projected/e37473ad-4cfc-4df2-9215-693d80954676-kube-api-access-8dh7x\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.855725 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.855659    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-home\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.956703 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.956616    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37473ad-4cfc-4df2-9215-693d80954676-tls-certs\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.956703 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.956656    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-tmp-dir\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.956703 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.956685    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-model-cache\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.957024 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.956730    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-dshm\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.957024 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.956756    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dh7x\" (UniqueName: \"kubernetes.io/projected/e37473ad-4cfc-4df2-9215-693d80954676-kube-api-access-8dh7x\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.957024 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.956790    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-home\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.957175 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.957111    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-home\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.957175 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.957151    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-tmp-dir\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.957254 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.957194    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-model-cache\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.958961 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.958935    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-dshm\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.959247 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.959230    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37473ad-4cfc-4df2-9215-693d80954676-tls-certs\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:37.964112 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:37.964094    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dh7x\" (UniqueName: \"kubernetes.io/projected/e37473ad-4cfc-4df2-9215-693d80954676-kube-api-access-8dh7x\") pod \"e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") " pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:38.009739 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:38.009709    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:38.280289 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:38.280261    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"]
Sep 17 23:26:38.282144 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:26:38.282121    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode37473ad_4cfc_4df2_9215_693d80954676.slice/crio-776a043739e6e24695617d7941e268eaeb27f014c915ca3cd1e553dcfa180776 WatchSource:0}: Error finding container 776a043739e6e24695617d7941e268eaeb27f014c915ca3cd1e553dcfa180776: Status 404 returned error can't find the container with id 776a043739e6e24695617d7941e268eaeb27f014c915ca3cd1e553dcfa180776
Sep 17 23:26:38.650966 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:38.650787    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:38.751025 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:38.750983    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:38.831947 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:38.831915    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7" event={"ID":"e37473ad-4cfc-4df2-9215-693d80954676","Type":"ContainerStarted","Data":"a25a8b44cf5a9b895ff25107e94f9dbb65e05912a1f85648860976bd1dade46b"}
Sep 17 23:26:38.832126 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:38.831959    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7" event={"ID":"e37473ad-4cfc-4df2-9215-693d80954676","Type":"ContainerStarted","Data":"776a043739e6e24695617d7941e268eaeb27f014c915ca3cd1e553dcfa180776"}
Sep 17 23:26:38.849229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:38.849175    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7" podStartSLOduration=1.849158909 podStartE2EDuration="1.849158909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:26:38.848342724 +0000 UTC m=+1197.072018472" watchObservedRunningTime="2026-09-17 23:26:38.849158909 +0000 UTC m=+1197.072834699"
Sep 17 23:26:41.350431 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:41.350384    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:41.601063 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:41.600931    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:42.161635 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:42.161603    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:42.470296 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:42.470169    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:43.351187 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:43.351151    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:45.138545 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.138514    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:45.185952 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.185924    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"]
Sep 17 23:26:45.190804 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.190776    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.194250 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.194226    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-09450b8d\"/\"default-dockercfg-969kl\""
Sep 17 23:26:45.194432 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.194415    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-09450b8d\"/\"e2e-lifecycle-model-09450b8d-kserve-self-signed-certs\""
Sep 17 23:26:45.194503 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.194466    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-09450b8d\"/\"kube-root-ca.crt\""
Sep 17 23:26:45.194550 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.194526    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-09450b8d\"/\"openshift-service-ca.crt\""
Sep 17 23:26:45.200784 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.200757    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"]
Sep 17 23:26:45.223138 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.223112    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:45.338790 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.338759    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-home\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.338981 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.338838    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzx64\" (UniqueName: \"kubernetes.io/projected/9a5295db-df86-4f52-a98f-f9149b276904-kube-api-access-fzx64\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.338981 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.338926    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5295db-df86-4f52-a98f-f9149b276904-tls-certs\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.339064 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.338984    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-tmp-dir\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.339064 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.339024    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-dshm\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.339064 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.339039    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-model-cache\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.440627 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.440541    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-tmp-dir\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.440627 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.440588    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-dshm\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.440627 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.440611    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-model-cache\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.440929 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.440652    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-home\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.440929 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.440727    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzx64\" (UniqueName: \"kubernetes.io/projected/9a5295db-df86-4f52-a98f-f9149b276904-kube-api-access-fzx64\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.440929 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.440774    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5295db-df86-4f52-a98f-f9149b276904-tls-certs\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.441106 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.441078    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-model-cache\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.441176 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.441121    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-tmp-dir\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.441176 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.441153    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-home\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.442845 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.442819    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-dshm\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.443118 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.443102    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5295db-df86-4f52-a98f-f9149b276904-tls-certs\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.448306 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.448287    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzx64\" (UniqueName: \"kubernetes.io/projected/9a5295db-df86-4f52-a98f-f9149b276904-kube-api-access-fzx64\") pod \"e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") " pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.501912 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.501874    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:45.776272 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.776241    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"]
Sep 17 23:26:45.778827 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:26:45.778801    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a5295db_df86_4f52_a98f_f9149b276904.slice/crio-8ef3064661a3a07613f81b10372e2f837e3575809c5e67bfb28b0880a782f984 WatchSource:0}: Error finding container 8ef3064661a3a07613f81b10372e2f837e3575809c5e67bfb28b0880a782f984: Status 404 returned error can't find the container with id 8ef3064661a3a07613f81b10372e2f837e3575809c5e67bfb28b0880a782f984
Sep 17 23:26:45.831506 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.831477    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:45.870919 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.870849    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm" event={"ID":"9a5295db-df86-4f52-a98f-f9149b276904","Type":"ContainerStarted","Data":"b69c3341d8040b569525207555b14a38e4272b07c4c6e7cc4d65fed9e6096702"}
Sep 17 23:26:45.871088 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.870926    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm" event={"ID":"9a5295db-df86-4f52-a98f-f9149b276904","Type":"ContainerStarted","Data":"8ef3064661a3a07613f81b10372e2f837e3575809c5e67bfb28b0880a782f984"}
Sep 17 23:26:45.895968 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.895915    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm" podStartSLOduration=0.895897014 podStartE2EDuration="895.897014ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:26:45.893943452 +0000 UTC m=+1204.117619201" watchObservedRunningTime="2026-09-17 23:26:45.895897014 +0000 UTC m=+1204.119572761"
Sep 17 23:26:45.925520 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:45.925489    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:46.740424 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:46.740389    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:47.585376 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:47.585345    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:48.010097 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:48.009991    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:48.010097 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:48.010050    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:48.022326 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:48.022304    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:48.431933 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:48.431901    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:48.896940 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:48.896913    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:26:49.118405 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.118377    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:49.821758 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.821725    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"]
Sep 17 23:26:49.825579 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.825555    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:26:49.827827 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.827800    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-f80c49-serving-cert\""
Sep 17 23:26:49.835885 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.835836    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"]
Sep 17 23:26:49.851488 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.851467    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:49.884329 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.884301    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7335ca2-98c3-4524-a488-8306e0042df8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:26:49.884485 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.884353    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7335ca2-98c3-4524-a488-8306e0042df8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:26:49.884485 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.884431    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gmcg\" (UniqueName: \"kubernetes.io/projected/e7335ca2-98c3-4524-a488-8306e0042df8-kube-api-access-8gmcg\") pod \"maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:26:49.884583 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.884480    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7335ca2-98c3-4524-a488-8306e0042df8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:26:49.917068 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.917042    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:49.985152 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.985105    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7335ca2-98c3-4524-a488-8306e0042df8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:26:49.985370 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.985265    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7335ca2-98c3-4524-a488-8306e0042df8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:26:49.985370 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.985358    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gmcg\" (UniqueName: \"kubernetes.io/projected/e7335ca2-98c3-4524-a488-8306e0042df8-kube-api-access-8gmcg\") pod \"maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:26:49.985508 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.985413    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7335ca2-98c3-4524-a488-8306e0042df8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:26:49.986053 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.986030    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7335ca2-98c3-4524-a488-8306e0042df8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:26:49.987706 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.987687    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7335ca2-98c3-4524-a488-8306e0042df8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:26:49.987803 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.987752    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7335ca2-98c3-4524-a488-8306e0042df8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:26:49.993303 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:49.993282    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gmcg\" (UniqueName: \"kubernetes.io/projected/e7335ca2-98c3-4524-a488-8306e0042df8-kube-api-access-8gmcg\") pod \"maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:26:50.019331 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:50.019305    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:50.139014 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:50.138932    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:26:50.316092 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:50.316041    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:50.414607 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:50.414580    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"]
Sep 17 23:26:50.417271 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:26:50.417236    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7335ca2_98c3_4524_a488_8306e0042df8.slice/crio-1f659cacb5bf62bdf277374168b9fafb4d100de24821abe01da11e703e0d26b4 WatchSource:0}: Error finding container 1f659cacb5bf62bdf277374168b9fafb4d100de24821abe01da11e703e0d26b4: Status 404 returned error can't find the container with id 1f659cacb5bf62bdf277374168b9fafb4d100de24821abe01da11e703e0d26b4
Sep 17 23:26:50.418497 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:50.418473    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:50.904347 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:50.904310    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls" event={"ID":"e7335ca2-98c3-4524-a488-8306e0042df8","Type":"ContainerStarted","Data":"2bebd88a66bf8b07e9b1fe6bb1c283911cdc6c934509e4720584018bb5cbde0d"}
Sep 17 23:26:50.904514 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:50.904388    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls" event={"ID":"e7335ca2-98c3-4524-a488-8306e0042df8","Type":"ContainerStarted","Data":"1f659cacb5bf62bdf277374168b9fafb4d100de24821abe01da11e703e0d26b4"}
Sep 17 23:26:50.904514 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:50.904431    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:26:50.921552 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:50.921503    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls" podStartSLOduration=1.553619908 podStartE2EDuration="1.921490995s" totalImagesPullingTime="367.871087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:26:50.919982161 +0000 UTC m=+1209.143657909" watchObservedRunningTime="2026-09-17 23:26:50.921490995 +0000 UTC m=+1209.145166743"
Sep 17 23:26:54.625981 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:54.625945    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:55.037175 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:55.037148    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:55.473748 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:55.473593    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:26:55.502545 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:55.502522    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:55.502778 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:55.502760    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:55.514812 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:55.514785    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:55.943393 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:55.943362    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:26:56.915187 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:26:56.915155    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:27:00.458455 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:00.458418    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"]
Sep 17 23:27:00.503759 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:00.503713    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:03.066297 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:03.066261    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:03.165740 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:03.165712    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:11.741244 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:11.741212    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:11.841074 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:11.841044    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:12.090319 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:12.090292    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:12.403916 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:12.403834    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:12.620358 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:12.620323    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:12.721769 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:12.721680    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:13.527881 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:13.527845    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:14.282520 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:14.282489    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:15.040307 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.040274    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:15.477951 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.477843    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d" podUID="bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f" containerName="main" containerID="cri-o://789c401bb125776ce656d3e1c1a39345d629232817c667de67ca4c2f59f7376c" gracePeriod=45
Sep 17 23:27:15.693470 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.693431    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:15.720144 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.720125    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:27:15.796935 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.796896    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:15.853731 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.853690    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-tmp-dir\") pod \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") "
Sep 17 23:27:15.853975 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.853769    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-tls-certs\") pod \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") "
Sep 17 23:27:15.853975 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.853800    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-dshm\") pod \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") "
Sep 17 23:27:15.853975 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.853888    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-home\") pod \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") "
Sep 17 23:27:15.853975 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.853927    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncmnb\" (UniqueName: \"kubernetes.io/projected/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-kube-api-access-ncmnb\") pod \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") "
Sep 17 23:27:15.854198 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.854021    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-model-cache\") pod \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\" (UID: \"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f\") "
Sep 17 23:27:15.854198 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.854034    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f" (UID: "bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:15.854198 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.854187    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-home" (OuterVolumeSpecName: "home") pod "bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f" (UID: "bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:15.854366 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.854341    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f" (UID: "bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:15.854466 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.854446    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-model-cache\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:27:15.854524 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.854490    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-tmp-dir\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:27:15.854524 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.854499    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-home\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:27:15.856244 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.856210    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-kube-api-access-ncmnb" (OuterVolumeSpecName: "kube-api-access-ncmnb") pod "bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f" (UID: "bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f"). InnerVolumeSpecName "kube-api-access-ncmnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:15.856480 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.856454    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-dshm" (OuterVolumeSpecName: "dshm") pod "bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f" (UID: "bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:15.856551 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.856494    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f" (UID: "bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:27:15.893609 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.893572    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:15.955117 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.955087    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-dshm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:27:15.955117 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.955113    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncmnb\" (UniqueName: \"kubernetes.io/projected/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-kube-api-access-ncmnb\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:27:15.955285 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:15.955123    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f-tls-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:27:16.034227 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:16.034195    2568 generic.go:356] "Generic (PLEG): container finished" podID="bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f" containerID="789c401bb125776ce656d3e1c1a39345d629232817c667de67ca4c2f59f7376c" exitCode=0
Sep 17 23:27:16.034393 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:16.034243    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"
Sep 17 23:27:16.034393 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:16.034282    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d" event={"ID":"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f","Type":"ContainerDied","Data":"789c401bb125776ce656d3e1c1a39345d629232817c667de67ca4c2f59f7376c"}
Sep 17 23:27:16.034393 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:16.034331    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d" event={"ID":"bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f","Type":"ContainerDied","Data":"c060037faaee9a86307d3c3d4c51fa8eaa2d6461a8884b15d5b9ca516b2b7f34"}
Sep 17 23:27:16.034393 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:16.034352    2568 scope.go:117] "RemoveContainer" containerID="789c401bb125776ce656d3e1c1a39345d629232817c667de67ca4c2f59f7376c"
Sep 17 23:27:16.044996 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:16.044976    2568 scope.go:117] "RemoveContainer" containerID="789c401bb125776ce656d3e1c1a39345d629232817c667de67ca4c2f59f7376c"
Sep 17 23:27:16.045325 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:27:16.045302    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789c401bb125776ce656d3e1c1a39345d629232817c667de67ca4c2f59f7376c\": container with ID starting with 789c401bb125776ce656d3e1c1a39345d629232817c667de67ca4c2f59f7376c not found: ID does not exist" containerID="789c401bb125776ce656d3e1c1a39345d629232817c667de67ca4c2f59f7376c"
Sep 17 23:27:16.045368 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:16.045336    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789c401bb125776ce656d3e1c1a39345d629232817c667de67ca4c2f59f7376c"} err="failed to get container status \"789c401bb125776ce656d3e1c1a39345d629232817c667de67ca4c2f59f7376c\": rpc error: code = NotFound desc = could not find container \"789c401bb125776ce656d3e1c1a39345d629232817c667de67ca4c2f59f7376c\": container with ID starting with 789c401bb125776ce656d3e1c1a39345d629232817c667de67ca4c2f59f7376c not found: ID does not exist"
Sep 17 23:27:16.063311 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:16.063250    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"]
Sep 17 23:27:16.066229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:16.066194    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-be3eb4/noten-model-be3eb4-kserve-74bf6698dc-qdc9d"]
Sep 17 23:27:16.635629 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:16.635594    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:17.524314 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:17.524276    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:17.626560 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:17.626528    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:17.726415 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:17.726382    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:17.991448 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:17.991376    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f" path="/var/lib/kubelet/pods/bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f/volumes"
Sep 17 23:27:18.020886 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:18.020839    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:18.115642 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:18.115613    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:18.437886 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:18.437842    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:19.454569 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:19.454527    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:20.157136 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:20.157093    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:20.937270 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:20.937238    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:21.644245 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:21.644211    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:24.484488 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.484454    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"]
Sep 17 23:27:24.484963 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.484944    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f" containerName="main"
Sep 17 23:27:24.485023 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.484966    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f" containerName="main"
Sep 17 23:27:24.485068 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.485047    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb2b0d85-9e2a-44cc-9f6a-3d9d5cd79c1f" containerName="main"
Sep 17 23:27:24.489517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.489500    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.492782 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.492743    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-736809\"/\"default-dockercfg-64l8t\""
Sep 17 23:27:24.492923 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.492743    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-736809\"/\"kube-root-ca.crt\""
Sep 17 23:27:24.492923 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.492790    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-736809\"/\"openshift-service-ca.crt\""
Sep 17 23:27:24.492923 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.492793    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-736809\"/\"distinct-2-w0-736809-kserve-self-signed-certs\""
Sep 17 23:27:24.498032 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.498009    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"]
Sep 17 23:27:24.538187 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.538162    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-tmp-dir\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.538331 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.538243    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29dcedab-f904-4ee6-ab58-ac399ab3de2b-tls-certs\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.538331 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.538262    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-dshm\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.538331 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.538282    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-model-cache\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.538331 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.538298    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnlvn\" (UniqueName: \"kubernetes.io/projected/29dcedab-f904-4ee6-ab58-ac399ab3de2b-kube-api-access-jnlvn\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.538469 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.538363    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-home\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.639286 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.639250    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29dcedab-f904-4ee6-ab58-ac399ab3de2b-tls-certs\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.639286 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.639289    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-dshm\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.639519 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.639313    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-model-cache\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.639519 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.639330    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnlvn\" (UniqueName: \"kubernetes.io/projected/29dcedab-f904-4ee6-ab58-ac399ab3de2b-kube-api-access-jnlvn\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.639519 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.639373    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-home\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.639519 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.639418    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-tmp-dir\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.639754 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.639728    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-model-cache\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.639801 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.639758    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-home\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.639843 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.639829    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-tmp-dir\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.641700 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.641682    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-dshm\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.642048 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.642023    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29dcedab-f904-4ee6-ab58-ac399ab3de2b-tls-certs\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.647418 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.647392    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnlvn\" (UniqueName: \"kubernetes.io/projected/29dcedab-f904-4ee6-ab58-ac399ab3de2b-kube-api-access-jnlvn\") pod \"distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") " pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.799381 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.799349    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:24.985805 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:24.985777    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:25.070335 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:25.070310    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"]
Sep 17 23:27:25.072457 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:27:25.072430    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29dcedab_f904_4ee6_ab58_ac399ab3de2b.slice/crio-3a2cb4ebda3d187ab352ac7971969376a91384daf7895de5af167b68c35e5634 WatchSource:0}: Error finding container 3a2cb4ebda3d187ab352ac7971969376a91384daf7895de5af167b68c35e5634: Status 404 returned error can't find the container with id 3a2cb4ebda3d187ab352ac7971969376a91384daf7895de5af167b68c35e5634
Sep 17 23:27:25.080247 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:25.080191    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj" event={"ID":"29dcedab-f904-4ee6-ab58-ac399ab3de2b","Type":"ContainerStarted","Data":"3a2cb4ebda3d187ab352ac7971969376a91384daf7895de5af167b68c35e5634"}
Sep 17 23:27:26.087306 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:26.087261    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj" event={"ID":"29dcedab-f904-4ee6-ab58-ac399ab3de2b","Type":"ContainerStarted","Data":"7ae44e29480b7102a969ab7c2d7ccf8e6be529ecdd014cd28c55ae536d8627f0"}
Sep 17 23:27:26.106201 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:26.106155    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj" podStartSLOduration=2.106139329 podStartE2EDuration="2.106139329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:26.105811606 +0000 UTC m=+1244.329487357" watchObservedRunningTime="2026-09-17 23:27:26.106139329 +0000 UTC m=+1244.329815123"
Sep 17 23:27:26.798002 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:26.797971    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:26.922064 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:26.922032    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:32.349922 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:32.349838    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:32.413412 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:32.413385    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:34.800109 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:34.800062    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:34.800109 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:34.800124    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:34.812626 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:34.812596    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:35.144360 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:35.144271    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:27:39.527604 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:39.527567    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:40.336495 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:40.336462    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:40.821443 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:40.821415    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:41.128981 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:41.128903    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:41.870693 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:41.870660    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:42.756468 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:42.756438    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:43.628300 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:43.628263    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:44.454302 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:44.454220    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:45.313606 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:45.313574    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:45.720139 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:45.720065    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:57.691376 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:57.691333    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"]
Sep 17 23:27:57.708982 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:27:57.708951    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:02.174608 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:02.174573    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:02.624932 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:02.624897    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:02.966608 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:02.966522    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:03.828110 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:03.828044    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:04.532673 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:04.532617    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:05.195118 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:05.195078    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:05.948038 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:05.948003    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:06.430713 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:06.430685    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:06.745046 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:06.744967    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:07.538490 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:07.538461    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:08.214769 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:08.214733    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:09.446846 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:09.446817    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:12.711368 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:12.711332    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7" podUID="e37473ad-4cfc-4df2-9215-693d80954676" containerName="main" containerID="cri-o://a25a8b44cf5a9b895ff25107e94f9dbb65e05912a1f85648860976bd1dade46b" gracePeriod=45
Sep 17 23:28:12.968634 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:12.968562    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:28:13.015023 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.014989    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37473ad-4cfc-4df2-9215-693d80954676-tls-certs\") pod \"e37473ad-4cfc-4df2-9215-693d80954676\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") "
Sep 17 23:28:13.015199 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.015100    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-dshm\") pod \"e37473ad-4cfc-4df2-9215-693d80954676\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") "
Sep 17 23:28:13.015199 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.015146    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-tmp-dir\") pod \"e37473ad-4cfc-4df2-9215-693d80954676\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") "
Sep 17 23:28:13.015199 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.015190    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-model-cache\") pod \"e37473ad-4cfc-4df2-9215-693d80954676\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") "
Sep 17 23:28:13.015370 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.015243    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dh7x\" (UniqueName: \"kubernetes.io/projected/e37473ad-4cfc-4df2-9215-693d80954676-kube-api-access-8dh7x\") pod \"e37473ad-4cfc-4df2-9215-693d80954676\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") "
Sep 17 23:28:13.015370 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.015315    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-home\") pod \"e37473ad-4cfc-4df2-9215-693d80954676\" (UID: \"e37473ad-4cfc-4df2-9215-693d80954676\") "
Sep 17 23:28:13.015482 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.015459    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e37473ad-4cfc-4df2-9215-693d80954676" (UID: "e37473ad-4cfc-4df2-9215-693d80954676"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:28:13.015545 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.015507    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-model-cache" (OuterVolumeSpecName: "model-cache") pod "e37473ad-4cfc-4df2-9215-693d80954676" (UID: "e37473ad-4cfc-4df2-9215-693d80954676"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:28:13.015688 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.015666    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-home" (OuterVolumeSpecName: "home") pod "e37473ad-4cfc-4df2-9215-693d80954676" (UID: "e37473ad-4cfc-4df2-9215-693d80954676"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:28:13.015830 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.015815    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-tmp-dir\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:13.015905 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.015835    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-model-cache\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:13.015905 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.015849    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-home\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:13.017539 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.017511    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-dshm" (OuterVolumeSpecName: "dshm") pod "e37473ad-4cfc-4df2-9215-693d80954676" (UID: "e37473ad-4cfc-4df2-9215-693d80954676"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:28:13.017826 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.017809    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e37473ad-4cfc-4df2-9215-693d80954676-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e37473ad-4cfc-4df2-9215-693d80954676" (UID: "e37473ad-4cfc-4df2-9215-693d80954676"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:28:13.017925 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.017831    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37473ad-4cfc-4df2-9215-693d80954676-kube-api-access-8dh7x" (OuterVolumeSpecName: "kube-api-access-8dh7x") pod "e37473ad-4cfc-4df2-9215-693d80954676" (UID: "e37473ad-4cfc-4df2-9215-693d80954676"). InnerVolumeSpecName "kube-api-access-8dh7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:28:13.116819 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.116772    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dh7x\" (UniqueName: \"kubernetes.io/projected/e37473ad-4cfc-4df2-9215-693d80954676-kube-api-access-8dh7x\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:13.116819 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.116809    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37473ad-4cfc-4df2-9215-693d80954676-tls-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:13.116819 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.116824    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e37473ad-4cfc-4df2-9215-693d80954676-dshm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:13.329857 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.329824    2568 generic.go:356] "Generic (PLEG): container finished" podID="e37473ad-4cfc-4df2-9215-693d80954676" containerID="a25a8b44cf5a9b895ff25107e94f9dbb65e05912a1f85648860976bd1dade46b" exitCode=0
Sep 17 23:28:13.330043 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.329895    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"
Sep 17 23:28:13.330043 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.329897    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7" event={"ID":"e37473ad-4cfc-4df2-9215-693d80954676","Type":"ContainerDied","Data":"a25a8b44cf5a9b895ff25107e94f9dbb65e05912a1f85648860976bd1dade46b"}
Sep 17 23:28:13.330043 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.330002    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7" event={"ID":"e37473ad-4cfc-4df2-9215-693d80954676","Type":"ContainerDied","Data":"776a043739e6e24695617d7941e268eaeb27f014c915ca3cd1e553dcfa180776"}
Sep 17 23:28:13.330043 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.330019    2568 scope.go:117] "RemoveContainer" containerID="a25a8b44cf5a9b895ff25107e94f9dbb65e05912a1f85648860976bd1dade46b"
Sep 17 23:28:13.339006 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.338989    2568 scope.go:117] "RemoveContainer" containerID="a25a8b44cf5a9b895ff25107e94f9dbb65e05912a1f85648860976bd1dade46b"
Sep 17 23:28:13.339275 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:28:13.339254    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25a8b44cf5a9b895ff25107e94f9dbb65e05912a1f85648860976bd1dade46b\": container with ID starting with a25a8b44cf5a9b895ff25107e94f9dbb65e05912a1f85648860976bd1dade46b not found: ID does not exist" containerID="a25a8b44cf5a9b895ff25107e94f9dbb65e05912a1f85648860976bd1dade46b"
Sep 17 23:28:13.339354 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.339287    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25a8b44cf5a9b895ff25107e94f9dbb65e05912a1f85648860976bd1dade46b"} err="failed to get container status \"a25a8b44cf5a9b895ff25107e94f9dbb65e05912a1f85648860976bd1dade46b\": rpc error: code = NotFound desc = could not find container \"a25a8b44cf5a9b895ff25107e94f9dbb65e05912a1f85648860976bd1dade46b\": container with ID starting with a25a8b44cf5a9b895ff25107e94f9dbb65e05912a1f85648860976bd1dade46b not found: ID does not exist"
Sep 17 23:28:13.352790 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.352768    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"]
Sep 17 23:28:13.355888 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.355856    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/e2e-conflict-b-3df45c8d-kserve-866ff8889b-qzlv7"]
Sep 17 23:28:13.990795 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:13.990754    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e37473ad-4cfc-4df2-9215-693d80954676" path="/var/lib/kubelet/pods/e37473ad-4cfc-4df2-9215-693d80954676/volumes"
Sep 17 23:28:15.925838 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:15.925809    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"]
Sep 17 23:28:15.926340 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:15.926310    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e37473ad-4cfc-4df2-9215-693d80954676" containerName="main"
Sep 17 23:28:15.926340 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:15.926323    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e37473ad-4cfc-4df2-9215-693d80954676" containerName="main"
Sep 17 23:28:15.926431 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:15.926390    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e37473ad-4cfc-4df2-9215-693d80954676" containerName="main"
Sep 17 23:28:15.931011 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:15.930993    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:15.933679 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:15.933659    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-825f30\"/\"distinct-w4-825f30-kserve-self-signed-certs\""
Sep 17 23:28:15.941397 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:15.941375    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"]
Sep 17 23:28:15.959670 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:15.959645    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:16.047228 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.047198    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-model-cache\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.047408 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.047244    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rkwv\" (UniqueName: \"kubernetes.io/projected/221ebd17-193c-4c22-9120-ebf7a4e4492e-kube-api-access-5rkwv\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.047408 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.047274    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-home\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.047408 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.047340    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-dshm\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.047544 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.047462    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-tmp-dir\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.047544 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.047500    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/221ebd17-193c-4c22-9120-ebf7a4e4492e-tls-certs\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.148205 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.148172    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-home\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.148380 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.148219    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-dshm\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.148380 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.148287    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-tmp-dir\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.148380 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.148317    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/221ebd17-193c-4c22-9120-ebf7a4e4492e-tls-certs\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.148380 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.148349    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-model-cache\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.148380 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.148378    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rkwv\" (UniqueName: \"kubernetes.io/projected/221ebd17-193c-4c22-9120-ebf7a4e4492e-kube-api-access-5rkwv\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.148631 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.148596    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-home\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.148693 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.148633    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-tmp-dir\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.148813 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.148790    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-model-cache\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.150556 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.150534    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-dshm\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.150698 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.150680    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/221ebd17-193c-4c22-9120-ebf7a4e4492e-tls-certs\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.156398 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.156376    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rkwv\" (UniqueName: \"kubernetes.io/projected/221ebd17-193c-4c22-9120-ebf7a4e4492e-kube-api-access-5rkwv\") pod \"distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.241444 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.241367    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:16.514471 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:16.514390    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"]
Sep 17 23:28:16.518126 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:28:16.518088    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod221ebd17_193c_4c22_9120_ebf7a4e4492e.slice/crio-79c80675aec3cad9ac25a506922feb7552610fdecebcaef77c21f93f8975b5b7 WatchSource:0}: Error finding container 79c80675aec3cad9ac25a506922feb7552610fdecebcaef77c21f93f8975b5b7: Status 404 returned error can't find the container with id 79c80675aec3cad9ac25a506922feb7552610fdecebcaef77c21f93f8975b5b7
Sep 17 23:28:17.354053 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.354015    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn" event={"ID":"221ebd17-193c-4c22-9120-ebf7a4e4492e","Type":"ContainerStarted","Data":"645b402d2adf84d098fa27b8e9fe627ead642cf16058c5ce614ce1dfa5e81400"}
Sep 17 23:28:17.354446 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.354066    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn" event={"ID":"221ebd17-193c-4c22-9120-ebf7a4e4492e","Type":"ContainerStarted","Data":"79c80675aec3cad9ac25a506922feb7552610fdecebcaef77c21f93f8975b5b7"}
Sep 17 23:28:17.626114 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.626007    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn" podStartSLOduration=2.625989153 podStartE2EDuration="2.625989153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:17.389573152 +0000 UTC m=+1295.613248900" watchObservedRunningTime="2026-09-17 23:28:17.625989153 +0000 UTC m=+1295.849664901"
Sep 17 23:28:17.627203 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.627181    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"]
Sep 17 23:28:17.627400 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.627381    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6" podUID="3fa4414e-4f92-40af-9f52-456fd8d4984a" containerName="maas-api" containerID="cri-o://458b26bfbeb68907e952d1af35e77b2c78cb3db0458883829854d0c37d9e38d6" gracePeriod=30
Sep 17 23:28:17.660172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.660136    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:17.760823 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.760790    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:17.863063 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.863031    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:17.875199 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.875179    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:28:17.960448 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.960369    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:17.969294 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.969271    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4414e-4f92-40af-9f52-456fd8d4984a-metrics-certs\") pod \"3fa4414e-4f92-40af-9f52-456fd8d4984a\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") "
Sep 17 23:28:17.969439 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.969315    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa4414e-4f92-40af-9f52-456fd8d4984a-maas-api-tls\") pod \"3fa4414e-4f92-40af-9f52-456fd8d4984a\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") "
Sep 17 23:28:17.969439 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.969368    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7rnm\" (UniqueName: \"kubernetes.io/projected/3fa4414e-4f92-40af-9f52-456fd8d4984a-kube-api-access-m7rnm\") pod \"3fa4414e-4f92-40af-9f52-456fd8d4984a\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") "
Sep 17 23:28:17.969525 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.969452    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa4414e-4f92-40af-9f52-456fd8d4984a-openshift-service-ca\") pod \"3fa4414e-4f92-40af-9f52-456fd8d4984a\" (UID: \"3fa4414e-4f92-40af-9f52-456fd8d4984a\") "
Sep 17 23:28:17.969921 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.969861    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa4414e-4f92-40af-9f52-456fd8d4984a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3fa4414e-4f92-40af-9f52-456fd8d4984a" (UID: "3fa4414e-4f92-40af-9f52-456fd8d4984a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:28:17.971424 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.971393    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa4414e-4f92-40af-9f52-456fd8d4984a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3fa4414e-4f92-40af-9f52-456fd8d4984a" (UID: "3fa4414e-4f92-40af-9f52-456fd8d4984a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:28:17.971424 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.971408    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa4414e-4f92-40af-9f52-456fd8d4984a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3fa4414e-4f92-40af-9f52-456fd8d4984a" (UID: "3fa4414e-4f92-40af-9f52-456fd8d4984a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:28:17.971581 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:17.971536    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa4414e-4f92-40af-9f52-456fd8d4984a-kube-api-access-m7rnm" (OuterVolumeSpecName: "kube-api-access-m7rnm") pod "3fa4414e-4f92-40af-9f52-456fd8d4984a" (UID: "3fa4414e-4f92-40af-9f52-456fd8d4984a"). InnerVolumeSpecName "kube-api-access-m7rnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:28:18.071002 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:18.070958    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa4414e-4f92-40af-9f52-456fd8d4984a-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:18.071002 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:18.071000    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4414e-4f92-40af-9f52-456fd8d4984a-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:18.071203 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:18.071013    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fa4414e-4f92-40af-9f52-456fd8d4984a-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:18.071203 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:18.071030    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7rnm\" (UniqueName: \"kubernetes.io/projected/3fa4414e-4f92-40af-9f52-456fd8d4984a-kube-api-access-m7rnm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:18.352751 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:18.352721    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:18.360356 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:18.360329    2568 generic.go:356] "Generic (PLEG): container finished" podID="3fa4414e-4f92-40af-9f52-456fd8d4984a" containerID="458b26bfbeb68907e952d1af35e77b2c78cb3db0458883829854d0c37d9e38d6" exitCode=0
Sep 17 23:28:18.360750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:18.360368    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"
Sep 17 23:28:18.360750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:18.360407    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6" event={"ID":"3fa4414e-4f92-40af-9f52-456fd8d4984a","Type":"ContainerDied","Data":"458b26bfbeb68907e952d1af35e77b2c78cb3db0458883829854d0c37d9e38d6"}
Sep 17 23:28:18.360750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:18.360452    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6" event={"ID":"3fa4414e-4f92-40af-9f52-456fd8d4984a","Type":"ContainerDied","Data":"d4b141446625529af2afc1c75bf936398e95f16e7fc4fca4b4aa93dbe9d7e510"}
Sep 17 23:28:18.360750 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:18.360474    2568 scope.go:117] "RemoveContainer" containerID="458b26bfbeb68907e952d1af35e77b2c78cb3db0458883829854d0c37d9e38d6"
Sep 17 23:28:18.369362 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:18.369346    2568 scope.go:117] "RemoveContainer" containerID="458b26bfbeb68907e952d1af35e77b2c78cb3db0458883829854d0c37d9e38d6"
Sep 17 23:28:18.369621 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:28:18.369601    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"458b26bfbeb68907e952d1af35e77b2c78cb3db0458883829854d0c37d9e38d6\": container with ID starting with 458b26bfbeb68907e952d1af35e77b2c78cb3db0458883829854d0c37d9e38d6 not found: ID does not exist" containerID="458b26bfbeb68907e952d1af35e77b2c78cb3db0458883829854d0c37d9e38d6"
Sep 17 23:28:18.369724 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:18.369629    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"458b26bfbeb68907e952d1af35e77b2c78cb3db0458883829854d0c37d9e38d6"} err="failed to get container status \"458b26bfbeb68907e952d1af35e77b2c78cb3db0458883829854d0c37d9e38d6\": rpc error: code = NotFound desc = could not find container \"458b26bfbeb68907e952d1af35e77b2c78cb3db0458883829854d0c37d9e38d6\": container with ID starting with 458b26bfbeb68907e952d1af35e77b2c78cb3db0458883829854d0c37d9e38d6 not found: ID does not exist"
Sep 17 23:28:18.378841 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:18.378812    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"]
Sep 17 23:28:18.389153 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:18.389121    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-09450b8d-ff5fb54fc-4l4p6"]
Sep 17 23:28:19.128587 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:19.128557    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:19.990681 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:19.990647    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fa4414e-4f92-40af-9f52-456fd8d4984a" path="/var/lib/kubelet/pods/3fa4414e-4f92-40af-9f52-456fd8d4984a/volumes"
Sep 17 23:28:20.027847 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:20.027819    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:20.733947 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:20.733910    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:20.983646 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:20.983608    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:21.418141 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:21.418101    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:22.135821 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:22.135792    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:22.803632 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:22.803600    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:25.966664 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:25.966631    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"]
Sep 17 23:28:26.012171 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:26.012142    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:26.242519 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:26.242432    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:26.242519 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:26.242487    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:26.254769 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:26.254744    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:26.414995 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:26.414969    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:28:40.600194 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.600162    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"]
Sep 17 23:28:40.600656 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.600570    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa4414e-4f92-40af-9f52-456fd8d4984a" containerName="maas-api"
Sep 17 23:28:40.600656 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.600580    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa4414e-4f92-40af-9f52-456fd8d4984a" containerName="maas-api"
Sep 17 23:28:40.600742 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.600667    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa4414e-4f92-40af-9f52-456fd8d4984a" containerName="maas-api"
Sep 17 23:28:40.603753 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.603732    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.606323 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.606296    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6ac6e\"/\"auth-test-model-f6ac6e-kserve-self-signed-certs\""
Sep 17 23:28:40.607667 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.607644    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6ac6e\"/\"default-dockercfg-5cvdt\""
Sep 17 23:28:40.607771 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.607702    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6ac6e\"/\"openshift-service-ca.crt\""
Sep 17 23:28:40.607771 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.607705    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6ac6e\"/\"kube-root-ca.crt\""
Sep 17 23:28:40.614734 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.614710    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"]
Sep 17 23:28:40.629800 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.629765    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:40.695107 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.695075    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-model-cache\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.695107 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.695113    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-tmp-dir\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.695326 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.695169    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-home\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.695326 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.695192    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8401988-f3c3-4357-8185-98fd7cab6dda-tls-certs\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.695326 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.695247    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbn87\" (UniqueName: \"kubernetes.io/projected/b8401988-f3c3-4357-8185-98fd7cab6dda-kube-api-access-dbn87\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.695326 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.695284    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-dshm\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.796274 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.796240    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbn87\" (UniqueName: \"kubernetes.io/projected/b8401988-f3c3-4357-8185-98fd7cab6dda-kube-api-access-dbn87\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.796461 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.796298    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-dshm\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.796461 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.796334    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-model-cache\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.796461 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.796354    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-tmp-dir\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.796461 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.796394    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-home\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.796664 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.796526    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8401988-f3c3-4357-8185-98fd7cab6dda-tls-certs\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.796766 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.796750    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-home\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.796829 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.796788    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-model-cache\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.796907 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.796829    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-tmp-dir\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.798709 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.798693    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-dshm\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.798892 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.798862    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8401988-f3c3-4357-8185-98fd7cab6dda-tls-certs\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.804240 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.804219    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbn87\" (UniqueName: \"kubernetes.io/projected/b8401988-f3c3-4357-8185-98fd7cab6dda-kube-api-access-dbn87\") pod \"auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.912702 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.912615    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:40.986339 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:40.986286    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm" podUID="9a5295db-df86-4f52-a98f-f9149b276904" containerName="main" containerID="cri-o://b69c3341d8040b569525207555b14a38e4272b07c4c6e7cc4d65fed9e6096702" gracePeriod=45
Sep 17 23:28:41.185364 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.185336    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"]
Sep 17 23:28:41.187167 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:28:41.187139    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8401988_f3c3_4357_8185_98fd7cab6dda.slice/crio-d9522837f7ed13253eeeffb31467ac1aaf6aa31bb18608507c67803931c45a6b WatchSource:0}: Error finding container d9522837f7ed13253eeeffb31467ac1aaf6aa31bb18608507c67803931c45a6b: Status 404 returned error can't find the container with id d9522837f7ed13253eeeffb31467ac1aaf6aa31bb18608507c67803931c45a6b
Sep 17 23:28:41.223752 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.223721    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:28:41.302491 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.302455    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzx64\" (UniqueName: \"kubernetes.io/projected/9a5295db-df86-4f52-a98f-f9149b276904-kube-api-access-fzx64\") pod \"9a5295db-df86-4f52-a98f-f9149b276904\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") "
Sep 17 23:28:41.302653 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.302534    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-home\") pod \"9a5295db-df86-4f52-a98f-f9149b276904\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") "
Sep 17 23:28:41.302653 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.302607    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5295db-df86-4f52-a98f-f9149b276904-tls-certs\") pod \"9a5295db-df86-4f52-a98f-f9149b276904\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") "
Sep 17 23:28:41.302794 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.302656    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-tmp-dir\") pod \"9a5295db-df86-4f52-a98f-f9149b276904\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") "
Sep 17 23:28:41.302794 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.302689    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-model-cache\") pod \"9a5295db-df86-4f52-a98f-f9149b276904\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") "
Sep 17 23:28:41.302794 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.302746    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-dshm\") pod \"9a5295db-df86-4f52-a98f-f9149b276904\" (UID: \"9a5295db-df86-4f52-a98f-f9149b276904\") "
Sep 17 23:28:41.303055 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.302782    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-home" (OuterVolumeSpecName: "home") pod "9a5295db-df86-4f52-a98f-f9149b276904" (UID: "9a5295db-df86-4f52-a98f-f9149b276904"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:28:41.303055 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.302924    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a5295db-df86-4f52-a98f-f9149b276904" (UID: "9a5295db-df86-4f52-a98f-f9149b276904"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:28:41.303055 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.302935    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9a5295db-df86-4f52-a98f-f9149b276904" (UID: "9a5295db-df86-4f52-a98f-f9149b276904"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:28:41.303252 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.303180    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-home\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:41.303252 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.303200    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-tmp-dir\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:41.303252 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.303213    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-model-cache\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:41.304793 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.304764    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5295db-df86-4f52-a98f-f9149b276904-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a5295db-df86-4f52-a98f-f9149b276904" (UID: "9a5295db-df86-4f52-a98f-f9149b276904"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:28:41.304915 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.304844    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-dshm" (OuterVolumeSpecName: "dshm") pod "9a5295db-df86-4f52-a98f-f9149b276904" (UID: "9a5295db-df86-4f52-a98f-f9149b276904"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:28:41.304973 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.304906    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5295db-df86-4f52-a98f-f9149b276904-kube-api-access-fzx64" (OuterVolumeSpecName: "kube-api-access-fzx64") pod "9a5295db-df86-4f52-a98f-f9149b276904" (UID: "9a5295db-df86-4f52-a98f-f9149b276904"). InnerVolumeSpecName "kube-api-access-fzx64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:28:41.404711 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.404670    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzx64\" (UniqueName: \"kubernetes.io/projected/9a5295db-df86-4f52-a98f-f9149b276904-kube-api-access-fzx64\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:41.404711 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.404701    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5295db-df86-4f52-a98f-f9149b276904-tls-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:41.404711 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.404719    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5295db-df86-4f52-a98f-f9149b276904-dshm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:28:41.482807 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.482720    2568 generic.go:356] "Generic (PLEG): container finished" podID="9a5295db-df86-4f52-a98f-f9149b276904" containerID="b69c3341d8040b569525207555b14a38e4272b07c4c6e7cc4d65fed9e6096702" exitCode=0
Sep 17 23:28:41.482807 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.482768    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"
Sep 17 23:28:41.483065 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.482806    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm" event={"ID":"9a5295db-df86-4f52-a98f-f9149b276904","Type":"ContainerDied","Data":"b69c3341d8040b569525207555b14a38e4272b07c4c6e7cc4d65fed9e6096702"}
Sep 17 23:28:41.483065 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.482855    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm" event={"ID":"9a5295db-df86-4f52-a98f-f9149b276904","Type":"ContainerDied","Data":"8ef3064661a3a07613f81b10372e2f837e3575809c5e67bfb28b0880a782f984"}
Sep 17 23:28:41.483065 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.482908    2568 scope.go:117] "RemoveContainer" containerID="b69c3341d8040b569525207555b14a38e4272b07c4c6e7cc4d65fed9e6096702"
Sep 17 23:28:41.484555 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.484533    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg" event={"ID":"b8401988-f3c3-4357-8185-98fd7cab6dda","Type":"ContainerStarted","Data":"a1fed6c4624c9d323340b3c904a1400f775463d82dc3bb62345e4b7c409975a9"}
Sep 17 23:28:41.484657 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.484568    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg" event={"ID":"b8401988-f3c3-4357-8185-98fd7cab6dda","Type":"ContainerStarted","Data":"d9522837f7ed13253eeeffb31467ac1aaf6aa31bb18608507c67803931c45a6b"}
Sep 17 23:28:41.492251 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.492233    2568 scope.go:117] "RemoveContainer" containerID="b69c3341d8040b569525207555b14a38e4272b07c4c6e7cc4d65fed9e6096702"
Sep 17 23:28:41.492499 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:28:41.492479    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69c3341d8040b569525207555b14a38e4272b07c4c6e7cc4d65fed9e6096702\": container with ID starting with b69c3341d8040b569525207555b14a38e4272b07c4c6e7cc4d65fed9e6096702 not found: ID does not exist" containerID="b69c3341d8040b569525207555b14a38e4272b07c4c6e7cc4d65fed9e6096702"
Sep 17 23:28:41.492563 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.492510    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69c3341d8040b569525207555b14a38e4272b07c4c6e7cc4d65fed9e6096702"} err="failed to get container status \"b69c3341d8040b569525207555b14a38e4272b07c4c6e7cc4d65fed9e6096702\": rpc error: code = NotFound desc = could not find container \"b69c3341d8040b569525207555b14a38e4272b07c4c6e7cc4d65fed9e6096702\": container with ID starting with b69c3341d8040b569525207555b14a38e4272b07c4c6e7cc4d65fed9e6096702 not found: ID does not exist"
Sep 17 23:28:41.505498 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.505459    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg" podStartSLOduration=1.505445948 podStartE2EDuration="1.505445948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:41.504319002 +0000 UTC m=+1319.727994770" watchObservedRunningTime="2026-09-17 23:28:41.505445948 +0000 UTC m=+1319.729121696"
Sep 17 23:28:41.517862 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.517830    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"]
Sep 17 23:28:41.521497 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.521475    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-09450b8d/e2e-lifecycle-model-09450b8d-kserve-6d49c6455f-77bnm"]
Sep 17 23:28:41.990496 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:41.990449    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a5295db-df86-4f52-a98f-f9149b276904" path="/var/lib/kubelet/pods/9a5295db-df86-4f52-a98f-f9149b276904/volumes"
Sep 17 23:28:50.913485 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:50.913439    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:50.913485 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:50.913493    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:50.926678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:50.926643    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:51.546362 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:51.546330    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:28:54.707027 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.706993    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"]
Sep 17 23:28:54.707430 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.707415    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5295db-df86-4f52-a98f-f9149b276904" containerName="main"
Sep 17 23:28:54.707481 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.707433    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5295db-df86-4f52-a98f-f9149b276904" containerName="main"
Sep 17 23:28:54.707516 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.707499    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5295db-df86-4f52-a98f-f9149b276904" containerName="main"
Sep 17 23:28:54.712368 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.712352    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.714761 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.714739    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-825f30\"/\"distinct-2-w4-825f30-kserve-self-signed-certs\""
Sep 17 23:28:54.719304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.719282    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"]
Sep 17 23:28:54.808375 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.808348    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:28:54.833338 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.833310    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-model-cache\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.833490 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.833355    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-dshm\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.833490 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.833397    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26631010-7003-4365-8935-004356931440-tls-certs\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.833490 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.833440    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdg95\" (UniqueName: \"kubernetes.io/projected/26631010-7003-4365-8935-004356931440-kube-api-access-mdg95\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.833490 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.833471    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-home\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.833490 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.833488    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-tmp-dir\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.934417 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.934384    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdg95\" (UniqueName: \"kubernetes.io/projected/26631010-7003-4365-8935-004356931440-kube-api-access-mdg95\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.934572 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.934435    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-home\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.934630 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.934608    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-tmp-dir\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.934770 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.934753    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-model-cache\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.934815 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.934801    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-home\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.934863 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.934815    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-dshm\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.934942 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.934889    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26631010-7003-4365-8935-004356931440-tls-certs\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.934996 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.934974    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-tmp-dir\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.935191 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.935168    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-model-cache\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.937074 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.937051    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-dshm\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.937206 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.937186    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26631010-7003-4365-8935-004356931440-tls-certs\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:54.942389 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:54.942367    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdg95\" (UniqueName: \"kubernetes.io/projected/26631010-7003-4365-8935-004356931440-kube-api-access-mdg95\") pod \"distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8\" (UID: \"26631010-7003-4365-8935-004356931440\") " pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:55.022277 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:55.022249    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:28:55.292435 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:55.292404    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"]
Sep 17 23:28:55.293593 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:28:55.293569    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26631010_7003_4365_8935_004356931440.slice/crio-38374220e2ab390663ecb68ac60baba7647e3cac4a53eb968bf2f95847c9ccd3 WatchSource:0}: Error finding container 38374220e2ab390663ecb68ac60baba7647e3cac4a53eb968bf2f95847c9ccd3: Status 404 returned error can't find the container with id 38374220e2ab390663ecb68ac60baba7647e3cac4a53eb968bf2f95847c9ccd3
Sep 17 23:28:55.557125 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:55.557037    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8" event={"ID":"26631010-7003-4365-8935-004356931440","Type":"ContainerStarted","Data":"e9cc4f35b7abc80684b230b00f184c9e00d8317b2f9bd957c001c6b301141782"}
Sep 17 23:28:55.557125 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:55.557082    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8" event={"ID":"26631010-7003-4365-8935-004356931440","Type":"ContainerStarted","Data":"38374220e2ab390663ecb68ac60baba7647e3cac4a53eb968bf2f95847c9ccd3"}
Sep 17 23:28:55.574662 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:55.574609    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8" podStartSLOduration=1.574591791 podStartE2EDuration="1.574591791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:55.572353759 +0000 UTC m=+1333.796029507" watchObservedRunningTime="2026-09-17 23:28:55.574591791 +0000 UTC m=+1333.798267538"
Sep 17 23:28:57.808456 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:28:57.808421    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:01.043484 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:01.043438    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:02.723417 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:02.723392    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:03.404265 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:03.404228    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:05.022541 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:05.022506    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:29:05.022541 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:05.022549    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:29:05.035055 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:05.035028    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:29:05.617034 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:05.617004    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:29:07.193576 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:07.193543    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:08.937024 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:08.936995    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:23.636799 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:23.636764    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:23.697850 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:23.697817    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:23.900327 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:23.900253    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:25.096261 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:25.096228    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:25.192492 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:25.192458    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:25.294687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:25.294656    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:25.853188 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:25.853160    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:25.956367 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:25.956333    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:26.052832 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:26.052789    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:27.272917 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:27.272885    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:32.941058 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:32.941022    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:33.040877 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:33.040842    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:33.581788 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:33.581753    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:39.284423 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:39.284387    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:39.386407 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:39.386364    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:39.484136 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:39.484103    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:48.151240 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:48.151202    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:49.958292 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:49.958247    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:50.165174 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:29:50.165140    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:11.168588 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:11.168560    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:17.667955 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:17.667920    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:17.769471 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:17.769437    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:17.996827 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:17.996744    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:18.283900 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:18.283849    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:18.492768 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:18.492731    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:20.649262 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:20.649228    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:21.212923 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:21.212889    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:21.308274 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:21.308219    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:21.641335 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:21.641303    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:22.447190 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:22.447145    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:23.235363 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:23.235330    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:24.128694 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:24.128658    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:24.935911 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:24.935859    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:25.682941 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:25.682907    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:26.619050 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:26.619014    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:26.725404 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:26.725371    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:26.823148 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:26.823101    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:27.160135 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:27.160105    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:27.261438 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:27.261401    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:45.348075 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:45.348039    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:46.004512 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.004474    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"]
Sep 17 23:30:46.008144 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.008120    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.011013 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.010983    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-736809\"/\"embedding-w0-736809-kserve-self-signed-certs\""
Sep 17 23:30:46.017511 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.017488    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"]
Sep 17 23:30:46.032671 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.032636    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:46.179218 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.179179    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7htjc\" (UniqueName: \"kubernetes.io/projected/b7502b72-6269-4fe2-8a26-12018c6367e7-kube-api-access-7htjc\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.179398 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.179227    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-home\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.179398 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.179290    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-model-cache\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.179398 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.179310    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-dshm\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.179398 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.179352    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7502b72-6269-4fe2-8a26-12018c6367e7-tls-certs\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.179543 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.179417    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-tmp-dir\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.280451 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.280418    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7htjc\" (UniqueName: \"kubernetes.io/projected/b7502b72-6269-4fe2-8a26-12018c6367e7-kube-api-access-7htjc\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.280635 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.280467    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-home\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.280635 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.280513    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-model-cache\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.280635 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.280535    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-dshm\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.280635 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.280552    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7502b72-6269-4fe2-8a26-12018c6367e7-tls-certs\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.280635 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.280571    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-tmp-dir\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.281038 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.281004    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-model-cache\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.281038 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.281026    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-home\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.281200 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.281083    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-tmp-dir\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.282979 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.282955    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-dshm\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.283229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.283208    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7502b72-6269-4fe2-8a26-12018c6367e7-tls-certs\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.291233 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.291213    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7htjc\" (UniqueName: \"kubernetes.io/projected/b7502b72-6269-4fe2-8a26-12018c6367e7-kube-api-access-7htjc\") pod \"embedding-w0-736809-kserve-65996558d9-plcvr\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") " pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.318040 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.318016    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:46.590642 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:46.590617    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"]
Sep 17 23:30:46.592773 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:30:46.592743    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7502b72_6269_4fe2_8a26_12018c6367e7.slice/crio-144d5911257b50907ad3e6edcf4304754afc0d46b35f3931479e99c759b409ab WatchSource:0}: Error finding container 144d5911257b50907ad3e6edcf4304754afc0d46b35f3931479e99c759b409ab: Status 404 returned error can't find the container with id 144d5911257b50907ad3e6edcf4304754afc0d46b35f3931479e99c759b409ab
Sep 17 23:30:47.116447 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:47.116416    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr" event={"ID":"b7502b72-6269-4fe2-8a26-12018c6367e7","Type":"ContainerStarted","Data":"c51eaefa9066b142d195855f192c8f52adb7e6b6af677f57653e2ef7e6f433a9"}
Sep 17 23:30:47.116447 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:47.116455    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr" event={"ID":"b7502b72-6269-4fe2-8a26-12018c6367e7","Type":"ContainerStarted","Data":"144d5911257b50907ad3e6edcf4304754afc0d46b35f3931479e99c759b409ab"}
Sep 17 23:30:47.134267 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:47.134134    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr" podStartSLOduration=2.134114226 podStartE2EDuration="2.134114226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:47.13350166 +0000 UTC m=+1445.357177418" watchObservedRunningTime="2026-09-17 23:30:47.134114226 +0000 UTC m=+1445.357789979"
Sep 17 23:30:47.577995 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:47.577965    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:48.828743 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:48.828706    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:49.646377 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:49.646347    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:50.310727 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:50.310700    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:50.970383 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:50.970352    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:51.274990 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:51.274962    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:51.710396 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:51.710312    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:51.935334 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:51.935296    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:52.007573 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:52.007484    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:52.284347 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:52.284306    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:52.525141 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:52.525082    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:53.229431 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:53.229399    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:54.120443 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:54.120413    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:54.952678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:54.952645    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:55.282195 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:55.282164    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:55.423955 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:55.423921    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:55.951577 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:55.951535    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:56.318221 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:56.318188    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:56.318616 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:56.318274    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:56.330686 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:56.330661    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:56.738416 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:56.738327    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:57.179059 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:57.179029    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:30:57.441421 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:57.441341    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:58.238495 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:58.238466    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:58.974618 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:58.974587    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:59.841206 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:30:59.841174    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:00.645141 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:00.645099    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:01.410487 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.410447    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"]
Sep 17 23:31:01.414960 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.414937    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:01.417361 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.417340    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b54c32-serving-cert\""
Sep 17 23:31:01.425218 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.424518    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"]
Sep 17 23:31:01.431965 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.431937    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d3af52-3619-488c-bd29-65084d670614-openshift-service-ca\") pod \"maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:01.432077 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.431977    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d3af52-3619-488c-bd29-65084d670614-maas-api-tls\") pod \"maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:01.432077 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.432027    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2d3af52-3619-488c-bd29-65084d670614-metrics-certs\") pod \"maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:01.432077 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.432043    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szdt5\" (UniqueName: \"kubernetes.io/projected/c2d3af52-3619-488c-bd29-65084d670614-kube-api-access-szdt5\") pod \"maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:01.444935 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.444911    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:01.512216 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.512178    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:01.532549 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.532523    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d3af52-3619-488c-bd29-65084d670614-openshift-service-ca\") pod \"maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:01.532715 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.532561    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d3af52-3619-488c-bd29-65084d670614-maas-api-tls\") pod \"maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:01.532715 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.532600    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2d3af52-3619-488c-bd29-65084d670614-metrics-certs\") pod \"maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:01.532715 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.532616    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szdt5\" (UniqueName: \"kubernetes.io/projected/c2d3af52-3619-488c-bd29-65084d670614-kube-api-access-szdt5\") pod \"maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:01.532901 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:31:01.532731    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-serving-cert: secret "maas-api-e2e-api-b-b54c32-serving-cert" not found
Sep 17 23:31:01.532901 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:31:01.532803    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d3af52-3619-488c-bd29-65084d670614-maas-api-tls podName:c2d3af52-3619-488c-bd29-65084d670614 nodeName:}" failed. No retries permitted until 2026-09-17 23:31:02.032787102 +0000 UTC m=+1460.256462829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2d3af52-3619-488c-bd29-65084d670614-maas-api-tls") pod "maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j" (UID: "c2d3af52-3619-488c-bd29-65084d670614") : secret "maas-api-e2e-api-b-b54c32-serving-cert" not found
Sep 17 23:31:01.533235 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.533205    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d3af52-3619-488c-bd29-65084d670614-openshift-service-ca\") pod \"maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:01.535050 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.535030    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2d3af52-3619-488c-bd29-65084d670614-metrics-certs\") pod \"maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:01.541101 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.541077    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szdt5\" (UniqueName: \"kubernetes.io/projected/c2d3af52-3619-488c-bd29-65084d670614-kube-api-access-szdt5\") pod \"maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:01.831768 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:01.831735    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:02.002977 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:02.002945    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:02.039020 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:02.038972    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d3af52-3619-488c-bd29-65084d670614-maas-api-tls\") pod \"maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:02.041354 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:02.041326    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d3af52-3619-488c-bd29-65084d670614-maas-api-tls\") pod \"maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:02.100861 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:02.100779    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:02.329034 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:02.328998    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:02.602515 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:02.602488    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"]
Sep 17 23:31:02.604143 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:31:02.604116    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d3af52_3619_488c_bd29_65084d670614.slice/crio-339a696e73c19df7758362bdfe811bdbd8f1ec9407f21bd6e3200bb7fabf109c WatchSource:0}: Error finding container 339a696e73c19df7758362bdfe811bdbd8f1ec9407f21bd6e3200bb7fabf109c: Status 404 returned error can't find the container with id 339a696e73c19df7758362bdfe811bdbd8f1ec9407f21bd6e3200bb7fabf109c
Sep 17 23:31:02.605615 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:02.605596    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:31:03.204775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:03.204670    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j" event={"ID":"c2d3af52-3619-488c-bd29-65084d670614","Type":"ContainerStarted","Data":"16d973ce1a036e23323e13759a4146dffbf706ffbff4022f9ea902d9c445596b"}
Sep 17 23:31:03.204775 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:03.204731    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j" event={"ID":"c2d3af52-3619-488c-bd29-65084d670614","Type":"ContainerStarted","Data":"339a696e73c19df7758362bdfe811bdbd8f1ec9407f21bd6e3200bb7fabf109c"}
Sep 17 23:31:03.205325 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:03.204895    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:03.224255 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:03.224197    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j" podStartSLOduration=1.948414279 podStartE2EDuration="2.22417905s" totalImagesPullingTime="275.764771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:31:03.222624111 +0000 UTC m=+1461.446299864" watchObservedRunningTime="2026-09-17 23:31:03.22417905 +0000 UTC m=+1461.447854826"
Sep 17 23:31:09.215143 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:09.215110    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:31:25.371920 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:25.371862    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:25.473604 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:25.473569    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:25.906292 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:25.906260    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:26.119654 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:26.119623    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:27.117126 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:27.117091    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:27.829026 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:27.828990    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:28.733517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:28.733486    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:29.641243 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:29.641213    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:30.347296 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:30.347261    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:31.031521 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:31.031491    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:31.850205 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:31.850166    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:32.526311 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:32.526278    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:33.606003 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:33.605960    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:34.029069 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:34.029043    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:34.793967 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:34.793938    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:43.667480 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:43.667447    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:43.746985 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:43.746953    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:43.816604 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:43.816565    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:43.886061 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:43.886031    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:44.227653 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:44.227623    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:44.328839 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:44.328806    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:45.107953 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:45.107920    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:45.837789 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:45.837755    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:46.752416 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:46.752378    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:47.454554 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:47.454514    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:48.354794 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:48.354754    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:49.023287 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:49.023256    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:49.832233 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:49.832203    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:50.124535 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:50.124450    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:50.221099 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:50.221052    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:50.382834 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:50.382751    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:51.558006 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:51.557962    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:52.347461 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:52.347426    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:53.045479 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:53.045444    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:53.843272 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:53.843237    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:54.656626 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:54.656594    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:55.323205 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:55.323138    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:56.135034 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:56.134991    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:56.843760 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:56.843728    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:58.437286 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:31:58.437245    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:06.260083 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.260048    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"]
Sep 17 23:32:06.260564 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.260257    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j" podUID="c2d3af52-3619-488c-bd29-65084d670614" containerName="maas-api" containerID="cri-o://16d973ce1a036e23323e13759a4146dffbf706ffbff4022f9ea902d9c445596b" gracePeriod=30
Sep 17 23:32:06.289252 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.289181    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:06.353845 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.353812    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:06.453501 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.453468    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:06.513530 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.513467    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:32:06.516287 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.516263    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:06.545825 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.545789    2568 generic.go:356] "Generic (PLEG): container finished" podID="c2d3af52-3619-488c-bd29-65084d670614" containerID="16d973ce1a036e23323e13759a4146dffbf706ffbff4022f9ea902d9c445596b" exitCode=0
Sep 17 23:32:06.546015 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.545831    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"
Sep 17 23:32:06.546015 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.545879    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j" event={"ID":"c2d3af52-3619-488c-bd29-65084d670614","Type":"ContainerDied","Data":"16d973ce1a036e23323e13759a4146dffbf706ffbff4022f9ea902d9c445596b"}
Sep 17 23:32:06.546015 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.545938    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j" event={"ID":"c2d3af52-3619-488c-bd29-65084d670614","Type":"ContainerDied","Data":"339a696e73c19df7758362bdfe811bdbd8f1ec9407f21bd6e3200bb7fabf109c"}
Sep 17 23:32:06.546015 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.545961    2568 scope.go:117] "RemoveContainer" containerID="16d973ce1a036e23323e13759a4146dffbf706ffbff4022f9ea902d9c445596b"
Sep 17 23:32:06.556632 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.556609    2568 scope.go:117] "RemoveContainer" containerID="16d973ce1a036e23323e13759a4146dffbf706ffbff4022f9ea902d9c445596b"
Sep 17 23:32:06.556915 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:32:06.556895    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d973ce1a036e23323e13759a4146dffbf706ffbff4022f9ea902d9c445596b\": container with ID starting with 16d973ce1a036e23323e13759a4146dffbf706ffbff4022f9ea902d9c445596b not found: ID does not exist" containerID="16d973ce1a036e23323e13759a4146dffbf706ffbff4022f9ea902d9c445596b"
Sep 17 23:32:06.556996 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.556923    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d973ce1a036e23323e13759a4146dffbf706ffbff4022f9ea902d9c445596b"} err="failed to get container status \"16d973ce1a036e23323e13759a4146dffbf706ffbff4022f9ea902d9c445596b\": rpc error: code = NotFound desc = could not find container \"16d973ce1a036e23323e13759a4146dffbf706ffbff4022f9ea902d9c445596b\": container with ID starting with 16d973ce1a036e23323e13759a4146dffbf706ffbff4022f9ea902d9c445596b not found: ID does not exist"
Sep 17 23:32:06.617460 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.617433    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szdt5\" (UniqueName: \"kubernetes.io/projected/c2d3af52-3619-488c-bd29-65084d670614-kube-api-access-szdt5\") pod \"c2d3af52-3619-488c-bd29-65084d670614\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") "
Sep 17 23:32:06.617614 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.617479    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2d3af52-3619-488c-bd29-65084d670614-metrics-certs\") pod \"c2d3af52-3619-488c-bd29-65084d670614\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") "
Sep 17 23:32:06.617614 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.617545    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d3af52-3619-488c-bd29-65084d670614-maas-api-tls\") pod \"c2d3af52-3619-488c-bd29-65084d670614\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") "
Sep 17 23:32:06.617614 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.617603    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d3af52-3619-488c-bd29-65084d670614-openshift-service-ca\") pod \"c2d3af52-3619-488c-bd29-65084d670614\" (UID: \"c2d3af52-3619-488c-bd29-65084d670614\") "
Sep 17 23:32:06.618091 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.618062    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d3af52-3619-488c-bd29-65084d670614-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2d3af52-3619-488c-bd29-65084d670614" (UID: "c2d3af52-3619-488c-bd29-65084d670614"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:32:06.619859 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.619830    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d3af52-3619-488c-bd29-65084d670614-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c2d3af52-3619-488c-bd29-65084d670614" (UID: "c2d3af52-3619-488c-bd29-65084d670614"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:06.619859 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.619830    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d3af52-3619-488c-bd29-65084d670614-kube-api-access-szdt5" (OuterVolumeSpecName: "kube-api-access-szdt5") pod "c2d3af52-3619-488c-bd29-65084d670614" (UID: "c2d3af52-3619-488c-bd29-65084d670614"). InnerVolumeSpecName "kube-api-access-szdt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:06.620046 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.619933    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d3af52-3619-488c-bd29-65084d670614-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2d3af52-3619-488c-bd29-65084d670614" (UID: "c2d3af52-3619-488c-bd29-65084d670614"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:06.719273 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.719234    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szdt5\" (UniqueName: \"kubernetes.io/projected/c2d3af52-3619-488c-bd29-65084d670614-kube-api-access-szdt5\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:06.719273 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.719267    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2d3af52-3619-488c-bd29-65084d670614-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:06.719273 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.719277    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d3af52-3619-488c-bd29-65084d670614-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:06.719273 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.719284    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d3af52-3619-488c-bd29-65084d670614-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:06.826163 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.826131    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:06.869950 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.869918    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"]
Sep 17 23:32:06.876495 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.876471    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b54c32-64f79d9ddd-jnv8j"]
Sep 17 23:32:06.937785 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:06.937761    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:07.626995 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:07.626950    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:07.991002 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:07.990926    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2d3af52-3619-488c-bd29-65084d670614" path="/var/lib/kubelet/pods/c2d3af52-3619-488c-bd29-65084d670614/volumes"
Sep 17 23:32:08.300544 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:08.300512    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:09.241822 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:09.241781    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:10.095496 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:10.095459    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:10.196283 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:10.196246    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:10.839627 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:10.839591    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:11.741382 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:11.741352    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:12.417279 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:12.417252    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:12.949816 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:12.949782    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:13.008389 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:13.008356    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:13.199812 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:13.199778    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:13.823743 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:13.823712    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:14.543731 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:14.543694    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:15.334191 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:15.334156    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:16.013986 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:16.013942    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:16.660960 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:16.660894    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:17.461688 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:17.461661    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:18.175711 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:18.175685    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:18.949446 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:18.949416    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:23.958992 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:23.958961    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"]
Sep 17 23:32:23.959379 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:23.959162    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl" podUID="9c2012bd-9df1-48e9-9c40-17e7e4f43eb0" containerName="maas-api" containerID="cri-o://67964f92433f62fb7fd3aa8f3c868ab12bfa3ea84358e8ac629f759fb007eb82" gracePeriod=30
Sep 17 23:32:23.982345 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:23.982318    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:24.046017 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.045967    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:24.106487 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.106460    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:24.168964 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.168929    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:24.207992 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.207967    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:32:24.230724 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.230654    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:24.322201 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.322171    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-openshift-service-ca\") pod \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") "
Sep 17 23:32:24.322373 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.322242    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-metrics-certs\") pod \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") "
Sep 17 23:32:24.322373 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.322365    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-maas-api-tls\") pod \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") "
Sep 17 23:32:24.322469 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.322426    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wlfx\" (UniqueName: \"kubernetes.io/projected/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-kube-api-access-6wlfx\") pod \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\" (UID: \"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0\") "
Sep 17 23:32:24.322581 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.322552    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c2012bd-9df1-48e9-9c40-17e7e4f43eb0" (UID: "9c2012bd-9df1-48e9-9c40-17e7e4f43eb0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:32:24.322814 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.322793    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:24.324445 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.324418    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9c2012bd-9df1-48e9-9c40-17e7e4f43eb0" (UID: "9c2012bd-9df1-48e9-9c40-17e7e4f43eb0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:24.324534 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.324506    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-kube-api-access-6wlfx" (OuterVolumeSpecName: "kube-api-access-6wlfx") pod "9c2012bd-9df1-48e9-9c40-17e7e4f43eb0" (UID: "9c2012bd-9df1-48e9-9c40-17e7e4f43eb0"). InnerVolumeSpecName "kube-api-access-6wlfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:24.324843 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.324820    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c2012bd-9df1-48e9-9c40-17e7e4f43eb0" (UID: "9c2012bd-9df1-48e9-9c40-17e7e4f43eb0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:24.423543 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.423502    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:24.423543 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.423537    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:24.423543 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.423547    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wlfx\" (UniqueName: \"kubernetes.io/projected/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0-kube-api-access-6wlfx\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:24.524824 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.524783    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:24.622192 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.622159    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:24.646409 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.646375    2568 generic.go:356] "Generic (PLEG): container finished" podID="9c2012bd-9df1-48e9-9c40-17e7e4f43eb0" containerID="67964f92433f62fb7fd3aa8f3c868ab12bfa3ea84358e8ac629f759fb007eb82" exitCode=0
Sep 17 23:32:24.646561 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.646418    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"
Sep 17 23:32:24.646561 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.646465    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl" event={"ID":"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0","Type":"ContainerDied","Data":"67964f92433f62fb7fd3aa8f3c868ab12bfa3ea84358e8ac629f759fb007eb82"}
Sep 17 23:32:24.646561 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.646516    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl" event={"ID":"9c2012bd-9df1-48e9-9c40-17e7e4f43eb0","Type":"ContainerDied","Data":"75f6da4fe602ce766e5d9b986d2ab0ee37dccf6d040cbcfadcb9b08606d94a8b"}
Sep 17 23:32:24.646561 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.646542    2568 scope.go:117] "RemoveContainer" containerID="67964f92433f62fb7fd3aa8f3c868ab12bfa3ea84358e8ac629f759fb007eb82"
Sep 17 23:32:24.658359 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.658342    2568 scope.go:117] "RemoveContainer" containerID="67964f92433f62fb7fd3aa8f3c868ab12bfa3ea84358e8ac629f759fb007eb82"
Sep 17 23:32:24.658622 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:32:24.658602    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67964f92433f62fb7fd3aa8f3c868ab12bfa3ea84358e8ac629f759fb007eb82\": container with ID starting with 67964f92433f62fb7fd3aa8f3c868ab12bfa3ea84358e8ac629f759fb007eb82 not found: ID does not exist" containerID="67964f92433f62fb7fd3aa8f3c868ab12bfa3ea84358e8ac629f759fb007eb82"
Sep 17 23:32:24.658702 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.658630    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67964f92433f62fb7fd3aa8f3c868ab12bfa3ea84358e8ac629f759fb007eb82"} err="failed to get container status \"67964f92433f62fb7fd3aa8f3c868ab12bfa3ea84358e8ac629f759fb007eb82\": rpc error: code = NotFound desc = could not find container \"67964f92433f62fb7fd3aa8f3c868ab12bfa3ea84358e8ac629f759fb007eb82\": container with ID starting with 67964f92433f62fb7fd3aa8f3c868ab12bfa3ea84358e8ac629f759fb007eb82 not found: ID does not exist"
Sep 17 23:32:24.669513 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.669493    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"]
Sep 17 23:32:24.677000 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:24.676981    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl"]
Sep 17 23:32:25.341918 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:25.341889    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:25.995440 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:25.995410    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c2012bd-9df1-48e9-9c40-17e7e4f43eb0" path="/var/lib/kubelet/pods/9c2012bd-9df1-48e9-9c40-17e7e4f43eb0/volumes"
Sep 17 23:32:26.057954 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:26.057928    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:26.848465 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:26.848424    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:27.133393 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:27.133287    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b52922-7f887f495f-ldrsl" podUID="9c2012bd-9df1-48e9-9c40-17e7e4f43eb0" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.50:8443/health\": context deadline exceeded"
Sep 17 23:32:27.853385 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:27.853353    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:28.617143 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:28.617110    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:30.288725 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:30.288691    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:30.389367 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:30.389338    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:30.526220 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:30.526184    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:30.903420 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:30.903389    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:31.006771 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:31.006736    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:31.727608 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:31.727576    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:32.642573 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:32.642542    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:33.279828 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:33.279795    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:33.380603 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:33.380564    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:34.143520 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:34.143481    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:34.939854 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:34.939822    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:35.736956 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:35.736923    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:35.884401 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:35.884364    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"]
Sep 17 23:32:35.889189 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:35.889161    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"]
Sep 17 23:32:35.903322 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:35.903288    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"]
Sep 17 23:32:35.913927 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:35.913900    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"]
Sep 17 23:32:35.962299 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:35.962267    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:36.509838 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:36.509803    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:37.324092 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:37.324061    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:38.015172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:38.015135    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:38.267198 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:38.267120    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"]
Sep 17 23:32:38.284059 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:38.284023    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"]
Sep 17 23:32:38.361474 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:38.361440    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:38.910175 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:38.910144    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:41.228992 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:41.228958    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:41.830812 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:41.830783    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:41.931100 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:41.931069    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:42.026275 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:42.026245    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:42.078205 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:42.078173    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:42.351116 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:42.351082    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:42.518755 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:42.518721    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:43.188528 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:43.188496    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:43.870564 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:43.870530    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:44.444670 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:44.444633    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"]
Sep 17 23:32:44.444955 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:44.444904    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6" podUID="71f11e1b-28fb-4b8b-b6af-198c5decffd4" containerName="maas-api" containerID="cri-o://1ee516df74bc9f506ba00cf516d5083264c317d1cb13cbff73508e32a44707ff" gracePeriod=30
Sep 17 23:32:44.478735 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:44.478701    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:44.561463 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:44.561430    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:44.627219 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:44.627188    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:44.700246 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:44.700161    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:44.895323 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:44.895296    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:32:44.899530 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:44.899502    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:45.035795 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.035757    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fntnn\" (UniqueName: \"kubernetes.io/projected/71f11e1b-28fb-4b8b-b6af-198c5decffd4-kube-api-access-fntnn\") pod \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") "
Sep 17 23:32:45.036022 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.035942    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f11e1b-28fb-4b8b-b6af-198c5decffd4-metrics-certs\") pod \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") "
Sep 17 23:32:45.036022 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.036007    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f11e1b-28fb-4b8b-b6af-198c5decffd4-maas-api-tls\") pod \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") "
Sep 17 23:32:45.036152 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.036051    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f11e1b-28fb-4b8b-b6af-198c5decffd4-openshift-service-ca\") pod \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\" (UID: \"71f11e1b-28fb-4b8b-b6af-198c5decffd4\") "
Sep 17 23:32:45.036500 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.036466    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f11e1b-28fb-4b8b-b6af-198c5decffd4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71f11e1b-28fb-4b8b-b6af-198c5decffd4" (UID: "71f11e1b-28fb-4b8b-b6af-198c5decffd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:32:45.038116 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.038092    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f11e1b-28fb-4b8b-b6af-198c5decffd4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71f11e1b-28fb-4b8b-b6af-198c5decffd4" (UID: "71f11e1b-28fb-4b8b-b6af-198c5decffd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:45.038192 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.038116    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f11e1b-28fb-4b8b-b6af-198c5decffd4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "71f11e1b-28fb-4b8b-b6af-198c5decffd4" (UID: "71f11e1b-28fb-4b8b-b6af-198c5decffd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:45.038192 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.038124    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f11e1b-28fb-4b8b-b6af-198c5decffd4-kube-api-access-fntnn" (OuterVolumeSpecName: "kube-api-access-fntnn") pod "71f11e1b-28fb-4b8b-b6af-198c5decffd4" (UID: "71f11e1b-28fb-4b8b-b6af-198c5decffd4"). InnerVolumeSpecName "kube-api-access-fntnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:45.054489 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.054463    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:45.137314 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.137282    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71f11e1b-28fb-4b8b-b6af-198c5decffd4-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:45.137314 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.137314    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71f11e1b-28fb-4b8b-b6af-198c5decffd4-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:45.137517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.137326    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f11e1b-28fb-4b8b-b6af-198c5decffd4-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:45.137517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.137339    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fntnn\" (UniqueName: \"kubernetes.io/projected/71f11e1b-28fb-4b8b-b6af-198c5decffd4-kube-api-access-fntnn\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:45.758094 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.758057    2568 generic.go:356] "Generic (PLEG): container finished" podID="71f11e1b-28fb-4b8b-b6af-198c5decffd4" containerID="1ee516df74bc9f506ba00cf516d5083264c317d1cb13cbff73508e32a44707ff" exitCode=0
Sep 17 23:32:45.758094 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.758098    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"
Sep 17 23:32:45.758439 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.758136    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6" event={"ID":"71f11e1b-28fb-4b8b-b6af-198c5decffd4","Type":"ContainerDied","Data":"1ee516df74bc9f506ba00cf516d5083264c317d1cb13cbff73508e32a44707ff"}
Sep 17 23:32:45.758439 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.758193    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6" event={"ID":"71f11e1b-28fb-4b8b-b6af-198c5decffd4","Type":"ContainerDied","Data":"4f8d029e55a72b19a19fb3a79d9459c2997822dd6b797891c5bef91c98c918f0"}
Sep 17 23:32:45.758439 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.758215    2568 scope.go:117] "RemoveContainer" containerID="1ee516df74bc9f506ba00cf516d5083264c317d1cb13cbff73508e32a44707ff"
Sep 17 23:32:45.767731 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.767714    2568 scope.go:117] "RemoveContainer" containerID="1ee516df74bc9f506ba00cf516d5083264c317d1cb13cbff73508e32a44707ff"
Sep 17 23:32:45.768051 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:32:45.768031    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee516df74bc9f506ba00cf516d5083264c317d1cb13cbff73508e32a44707ff\": container with ID starting with 1ee516df74bc9f506ba00cf516d5083264c317d1cb13cbff73508e32a44707ff not found: ID does not exist" containerID="1ee516df74bc9f506ba00cf516d5083264c317d1cb13cbff73508e32a44707ff"
Sep 17 23:32:45.768138 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.768062    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee516df74bc9f506ba00cf516d5083264c317d1cb13cbff73508e32a44707ff"} err="failed to get container status \"1ee516df74bc9f506ba00cf516d5083264c317d1cb13cbff73508e32a44707ff\": rpc error: code = NotFound desc = could not find container \"1ee516df74bc9f506ba00cf516d5083264c317d1cb13cbff73508e32a44707ff\": container with ID starting with 1ee516df74bc9f506ba00cf516d5083264c317d1cb13cbff73508e32a44707ff not found: ID does not exist"
Sep 17 23:32:45.779561 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.779532    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"]
Sep 17 23:32:45.781413 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.781395    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-825f30-5448495788-zbxs6"]
Sep 17 23:32:45.918505 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.918471    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:45.990242 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:45.990211    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71f11e1b-28fb-4b8b-b6af-198c5decffd4" path="/var/lib/kubelet/pods/71f11e1b-28fb-4b8b-b6af-198c5decffd4/volumes"
Sep 17 23:32:46.614575 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:46.614541    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:47.517229 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:47.517194    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:48.499854 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:48.499822    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:48.600101 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:48.600071    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:48.749041 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:48.749007    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:50.622596 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:50.622565    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"]
Sep 17 23:32:50.623044 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:50.622829    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk" podUID="7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a" containerName="maas-api" containerID="cri-o://b19c93b09400d67014dba5e6fdbc2c513d1afd6ab4b5c764748e5a6697761863" gracePeriod=30
Sep 17 23:32:50.648740 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:50.648711    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:50.721901 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:50.721859    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:50.786121 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:50.786089    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:50.862394 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:50.862362    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:50.904399 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:50.904369    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8" podUID="26631010-7003-4365-8935-004356931440" containerName="main" containerID="cri-o://e9cc4f35b7abc80684b230b00f184c9e00d8317b2f9bd957c001c6b301141782" gracePeriod=45
Sep 17 23:32:50.909720 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:50.909690    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn" podUID="221ebd17-193c-4c22-9120-ebf7a4e4492e" containerName="main" containerID="cri-o://645b402d2adf84d098fa27b8e9fe627ead642cf16058c5ce614ce1dfa5e81400" gracePeriod=45
Sep 17 23:32:50.923210 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:50.923182    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52" podUID="6d35d4f5-288e-4f35-b646-d38e0c03ebf0" containerName="main" containerID="cri-o://9faf8e02d890277582e13db10e4df0c8682ca49b8aa6a86332fdb99657afafd5" gracePeriod=45
Sep 17 23:32:50.924021 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:50.923995    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:50.933027 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:50.932999    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs" podUID="3028dec1-b28d-4853-a9fb-ab9578fbc4ba" containerName="main" containerID="cri-o://18d4d9bfc6b3e3b517501c169e0dda4e30b0fa310d66d906d0f3d4b0824c5d40" gracePeriod=45
Sep 17 23:32:51.156348 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.156270    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:51.223696 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.223667    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:32:51.225354 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.225332    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:32:51.263410 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.262823    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:32:51.268523 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.268503    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:32:51.271853 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.271826    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:32:51.279968 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.279921    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:51.298502 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.298472    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-model-cache\") pod \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") "
Sep 17 23:32:51.298730 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.298520    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-tls-certs\") pod \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") "
Sep 17 23:32:51.298730 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.298572    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-metrics-certs\") pod \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") "
Sep 17 23:32:51.298730 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.298603    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-486vn\" (UniqueName: \"kubernetes.io/projected/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-kube-api-access-486vn\") pod \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") "
Sep 17 23:32:51.298730 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.298663    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-home\") pod \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") "
Sep 17 23:32:51.298730 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.298700    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4hrn\" (UniqueName: \"kubernetes.io/projected/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-kube-api-access-x4hrn\") pod \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") "
Sep 17 23:32:51.299159 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.298741    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-tmp-dir\") pod \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") "
Sep 17 23:32:51.299159 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.298774    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d35d4f5-288e-4f35-b646-d38e0c03ebf0" (UID: "6d35d4f5-288e-4f35-b646-d38e0c03ebf0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.299159 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.298795    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-openshift-service-ca\") pod \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") "
Sep 17 23:32:51.299159 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.298880    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-maas-api-tls\") pod \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\" (UID: \"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a\") "
Sep 17 23:32:51.299159 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.298918    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-dshm\") pod \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\" (UID: \"6d35d4f5-288e-4f35-b646-d38e0c03ebf0\") "
Sep 17 23:32:51.299431 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.299252    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-model-cache\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.300190 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.299914    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-home" (OuterVolumeSpecName: "home") pod "6d35d4f5-288e-4f35-b646-d38e0c03ebf0" (UID: "6d35d4f5-288e-4f35-b646-d38e0c03ebf0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.300667 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.300645    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a" (UID: "7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:32:51.301069 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.301048    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d35d4f5-288e-4f35-b646-d38e0c03ebf0" (UID: "6d35d4f5-288e-4f35-b646-d38e0c03ebf0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.302027 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.302003    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-dshm" (OuterVolumeSpecName: "dshm") pod "6d35d4f5-288e-4f35-b646-d38e0c03ebf0" (UID: "6d35d4f5-288e-4f35-b646-d38e0c03ebf0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.302283 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.302252    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-kube-api-access-486vn" (OuterVolumeSpecName: "kube-api-access-486vn") pod "6d35d4f5-288e-4f35-b646-d38e0c03ebf0" (UID: "6d35d4f5-288e-4f35-b646-d38e0c03ebf0"). InnerVolumeSpecName "kube-api-access-486vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:51.302490 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.302443    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-kube-api-access-x4hrn" (OuterVolumeSpecName: "kube-api-access-x4hrn") pod "7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a" (UID: "7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a"). InnerVolumeSpecName "kube-api-access-x4hrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:51.302633 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.302530    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d35d4f5-288e-4f35-b646-d38e0c03ebf0" (UID: "6d35d4f5-288e-4f35-b646-d38e0c03ebf0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:51.303921 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.303891    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a" (UID: "7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:51.304454 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.304432    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a" (UID: "7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:51.370151 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.370118    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:51.399700 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.399671    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-model-cache\") pod \"221ebd17-193c-4c22-9120-ebf7a4e4492e\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") "
Sep 17 23:32:51.399858 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.399733    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-model-cache\") pod \"26631010-7003-4365-8935-004356931440\" (UID: \"26631010-7003-4365-8935-004356931440\") "
Sep 17 23:32:51.399858 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.399765    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-home\") pod \"221ebd17-193c-4c22-9120-ebf7a4e4492e\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") "
Sep 17 23:32:51.399858 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.399789    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdg95\" (UniqueName: \"kubernetes.io/projected/26631010-7003-4365-8935-004356931440-kube-api-access-mdg95\") pod \"26631010-7003-4365-8935-004356931440\" (UID: \"26631010-7003-4365-8935-004356931440\") "
Sep 17 23:32:51.399858 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.399813    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-tls-certs\") pod \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") "
Sep 17 23:32:51.400136 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.399858    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-model-cache\") pod \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") "
Sep 17 23:32:51.400192 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400146    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-tmp-dir\") pod \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") "
Sep 17 23:32:51.400192 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400183    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-tmp-dir\") pod \"221ebd17-193c-4c22-9120-ebf7a4e4492e\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") "
Sep 17 23:32:51.400272 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400230    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-tmp-dir\") pod \"26631010-7003-4365-8935-004356931440\" (UID: \"26631010-7003-4365-8935-004356931440\") "
Sep 17 23:32:51.400272 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400059    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-model-cache" (OuterVolumeSpecName: "model-cache") pod "221ebd17-193c-4c22-9120-ebf7a4e4492e" (UID: "221ebd17-193c-4c22-9120-ebf7a4e4492e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.400358 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400254    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-home" (OuterVolumeSpecName: "home") pod "221ebd17-193c-4c22-9120-ebf7a4e4492e" (UID: "221ebd17-193c-4c22-9120-ebf7a4e4492e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.400358 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400262    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "3028dec1-b28d-4853-a9fb-ab9578fbc4ba" (UID: "3028dec1-b28d-4853-a9fb-ab9578fbc4ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.400358 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400266    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26631010-7003-4365-8935-004356931440-tls-certs\") pod \"26631010-7003-4365-8935-004356931440\" (UID: \"26631010-7003-4365-8935-004356931440\") "
Sep 17 23:32:51.400467 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400362    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3028dec1-b28d-4853-a9fb-ab9578fbc4ba" (UID: "3028dec1-b28d-4853-a9fb-ab9578fbc4ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.400467 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400402    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-dshm\") pod \"26631010-7003-4365-8935-004356931440\" (UID: \"26631010-7003-4365-8935-004356931440\") "
Sep 17 23:32:51.400467 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400436    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "221ebd17-193c-4c22-9120-ebf7a4e4492e" (UID: "221ebd17-193c-4c22-9120-ebf7a4e4492e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.400467 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400442    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/221ebd17-193c-4c22-9120-ebf7a4e4492e-tls-certs\") pod \"221ebd17-193c-4c22-9120-ebf7a4e4492e\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") "
Sep 17 23:32:51.400673 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400499    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rkwv\" (UniqueName: \"kubernetes.io/projected/221ebd17-193c-4c22-9120-ebf7a4e4492e-kube-api-access-5rkwv\") pod \"221ebd17-193c-4c22-9120-ebf7a4e4492e\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") "
Sep 17 23:32:51.400673 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400451    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26631010-7003-4365-8935-004356931440" (UID: "26631010-7003-4365-8935-004356931440"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.400673 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400462    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-model-cache" (OuterVolumeSpecName: "model-cache") pod "26631010-7003-4365-8935-004356931440" (UID: "26631010-7003-4365-8935-004356931440"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.400673 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400533    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-home\") pod \"26631010-7003-4365-8935-004356931440\" (UID: \"26631010-7003-4365-8935-004356931440\") "
Sep 17 23:32:51.400673 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400596    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-dshm\") pod \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") "
Sep 17 23:32:51.400948 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400680    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-dshm\") pod \"221ebd17-193c-4c22-9120-ebf7a4e4492e\" (UID: \"221ebd17-193c-4c22-9120-ebf7a4e4492e\") "
Sep 17 23:32:51.400948 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400719    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2brp\" (UniqueName: \"kubernetes.io/projected/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-kube-api-access-r2brp\") pod \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") "
Sep 17 23:32:51.400948 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400738    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-home" (OuterVolumeSpecName: "home") pod "26631010-7003-4365-8935-004356931440" (UID: "26631010-7003-4365-8935-004356931440"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.400948 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.400756    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-home\") pod \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\" (UID: \"3028dec1-b28d-4853-a9fb-ab9578fbc4ba\") "
Sep 17 23:32:51.401202 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401169    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-tmp-dir\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401202 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401196    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-tls-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401212    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401226    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-486vn\" (UniqueName: \"kubernetes.io/projected/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-kube-api-access-486vn\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401239    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-home\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401253    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-home\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401268    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4hrn\" (UniqueName: \"kubernetes.io/projected/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-kube-api-access-x4hrn\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401281    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-tmp-dir\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401297    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-model-cache\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401312    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401326    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-model-cache\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401794 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401339    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-home\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401794 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401354    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401794 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401366    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-model-cache\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401794 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401378    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-tmp-dir\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401794 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401393    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-tmp-dir\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.401794 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401405    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d35d4f5-288e-4f35-b646-d38e0c03ebf0-dshm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.402724 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.401849    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-home" (OuterVolumeSpecName: "home") pod "3028dec1-b28d-4853-a9fb-ab9578fbc4ba" (UID: "3028dec1-b28d-4853-a9fb-ab9578fbc4ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.403204 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.403174    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-dshm" (OuterVolumeSpecName: "dshm") pod "26631010-7003-4365-8935-004356931440" (UID: "26631010-7003-4365-8935-004356931440"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.403409 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.403327    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26631010-7003-4365-8935-004356931440-kube-api-access-mdg95" (OuterVolumeSpecName: "kube-api-access-mdg95") pod "26631010-7003-4365-8935-004356931440" (UID: "26631010-7003-4365-8935-004356931440"). InnerVolumeSpecName "kube-api-access-mdg95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:51.403409 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.403391    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221ebd17-193c-4c22-9120-ebf7a4e4492e-kube-api-access-5rkwv" (OuterVolumeSpecName: "kube-api-access-5rkwv") pod "221ebd17-193c-4c22-9120-ebf7a4e4492e" (UID: "221ebd17-193c-4c22-9120-ebf7a4e4492e"). InnerVolumeSpecName "kube-api-access-5rkwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:51.403569 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.403532    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-dshm" (OuterVolumeSpecName: "dshm") pod "221ebd17-193c-4c22-9120-ebf7a4e4492e" (UID: "221ebd17-193c-4c22-9120-ebf7a4e4492e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.403644 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.403622    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-dshm" (OuterVolumeSpecName: "dshm") pod "3028dec1-b28d-4853-a9fb-ab9578fbc4ba" (UID: "3028dec1-b28d-4853-a9fb-ab9578fbc4ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:51.404388 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.404366    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3028dec1-b28d-4853-a9fb-ab9578fbc4ba" (UID: "3028dec1-b28d-4853-a9fb-ab9578fbc4ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:51.404548 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.404529    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/221ebd17-193c-4c22-9120-ebf7a4e4492e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "221ebd17-193c-4c22-9120-ebf7a4e4492e" (UID: "221ebd17-193c-4c22-9120-ebf7a4e4492e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:51.404597 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.404581    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26631010-7003-4365-8935-004356931440-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26631010-7003-4365-8935-004356931440" (UID: "26631010-7003-4365-8935-004356931440"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:51.405308 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.405235    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-kube-api-access-r2brp" (OuterVolumeSpecName: "kube-api-access-r2brp") pod "3028dec1-b28d-4853-a9fb-ab9578fbc4ba" (UID: "3028dec1-b28d-4853-a9fb-ab9578fbc4ba"). InnerVolumeSpecName "kube-api-access-r2brp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:51.436575 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.436550    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:51.501983 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.501947    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26631010-7003-4365-8935-004356931440-tls-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.501983 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.501976    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26631010-7003-4365-8935-004356931440-dshm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.501983 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.501985    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/221ebd17-193c-4c22-9120-ebf7a4e4492e-tls-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.502208 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.501994    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rkwv\" (UniqueName: \"kubernetes.io/projected/221ebd17-193c-4c22-9120-ebf7a4e4492e-kube-api-access-5rkwv\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.502208 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.502002    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-dshm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.502208 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.502011    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/221ebd17-193c-4c22-9120-ebf7a4e4492e-dshm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.502208 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.502019    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2brp\" (UniqueName: \"kubernetes.io/projected/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-kube-api-access-r2brp\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.502208 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.502026    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-home\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.502208 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.502034    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdg95\" (UniqueName: \"kubernetes.io/projected/26631010-7003-4365-8935-004356931440-kube-api-access-mdg95\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.502208 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.502055    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3028dec1-b28d-4853-a9fb-ab9578fbc4ba-tls-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:51.790887 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.790837    2568 generic.go:356] "Generic (PLEG): container finished" podID="221ebd17-193c-4c22-9120-ebf7a4e4492e" containerID="645b402d2adf84d098fa27b8e9fe627ead642cf16058c5ce614ce1dfa5e81400" exitCode=0
Sep 17 23:32:51.791352 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.790912    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"
Sep 17 23:32:51.791352 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.790931    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn" event={"ID":"221ebd17-193c-4c22-9120-ebf7a4e4492e","Type":"ContainerDied","Data":"645b402d2adf84d098fa27b8e9fe627ead642cf16058c5ce614ce1dfa5e81400"}
Sep 17 23:32:51.791352 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.790970    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn" event={"ID":"221ebd17-193c-4c22-9120-ebf7a4e4492e","Type":"ContainerDied","Data":"79c80675aec3cad9ac25a506922feb7552610fdecebcaef77c21f93f8975b5b7"}
Sep 17 23:32:51.791352 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.790992    2568 scope.go:117] "RemoveContainer" containerID="645b402d2adf84d098fa27b8e9fe627ead642cf16058c5ce614ce1dfa5e81400"
Sep 17 23:32:51.792500 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.792476    2568 generic.go:356] "Generic (PLEG): container finished" podID="7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a" containerID="b19c93b09400d67014dba5e6fdbc2c513d1afd6ab4b5c764748e5a6697761863" exitCode=0
Sep 17 23:32:51.792605 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.792581    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"
Sep 17 23:32:51.792677 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.792621    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk" event={"ID":"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a","Type":"ContainerDied","Data":"b19c93b09400d67014dba5e6fdbc2c513d1afd6ab4b5c764748e5a6697761863"}
Sep 17 23:32:51.792729 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.792673    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk" event={"ID":"7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a","Type":"ContainerDied","Data":"f78313ec68a8c314d99c66277f12f6a32afe76fb30542d06f1d2577f0594f54f"}
Sep 17 23:32:51.794211 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.794180    2568 generic.go:356] "Generic (PLEG): container finished" podID="3028dec1-b28d-4853-a9fb-ab9578fbc4ba" containerID="18d4d9bfc6b3e3b517501c169e0dda4e30b0fa310d66d906d0f3d4b0824c5d40" exitCode=0
Sep 17 23:32:51.794324 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.794214    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"
Sep 17 23:32:51.794324 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.794231    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs" event={"ID":"3028dec1-b28d-4853-a9fb-ab9578fbc4ba","Type":"ContainerDied","Data":"18d4d9bfc6b3e3b517501c169e0dda4e30b0fa310d66d906d0f3d4b0824c5d40"}
Sep 17 23:32:51.794324 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.794268    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs" event={"ID":"3028dec1-b28d-4853-a9fb-ab9578fbc4ba","Type":"ContainerDied","Data":"5803869c092dd7b3de44e7e1b2075ed18545ecfc6546d21086ca6582c8a16b47"}
Sep 17 23:32:51.795788 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.795765    2568 generic.go:356] "Generic (PLEG): container finished" podID="26631010-7003-4365-8935-004356931440" containerID="e9cc4f35b7abc80684b230b00f184c9e00d8317b2f9bd957c001c6b301141782" exitCode=0
Sep 17 23:32:51.795919 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.795821    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"
Sep 17 23:32:51.795995 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.795825    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8" event={"ID":"26631010-7003-4365-8935-004356931440","Type":"ContainerDied","Data":"e9cc4f35b7abc80684b230b00f184c9e00d8317b2f9bd957c001c6b301141782"}
Sep 17 23:32:51.796060 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.796006    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8" event={"ID":"26631010-7003-4365-8935-004356931440","Type":"ContainerDied","Data":"38374220e2ab390663ecb68ac60baba7647e3cac4a53eb968bf2f95847c9ccd3"}
Sep 17 23:32:51.797381 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.797359    2568 generic.go:356] "Generic (PLEG): container finished" podID="6d35d4f5-288e-4f35-b646-d38e0c03ebf0" containerID="9faf8e02d890277582e13db10e4df0c8682ca49b8aa6a86332fdb99657afafd5" exitCode=0
Sep 17 23:32:51.797485 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.797394    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52" event={"ID":"6d35d4f5-288e-4f35-b646-d38e0c03ebf0","Type":"ContainerDied","Data":"9faf8e02d890277582e13db10e4df0c8682ca49b8aa6a86332fdb99657afafd5"}
Sep 17 23:32:51.797485 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.797419    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52" event={"ID":"6d35d4f5-288e-4f35-b646-d38e0c03ebf0","Type":"ContainerDied","Data":"a1d05c9aa9a71d800bbc99da8a6522f9f5353ceb87611db23bdd77097019c946"}
Sep 17 23:32:51.797485 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.797474    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"
Sep 17 23:32:51.801276 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.801258    2568 scope.go:117] "RemoveContainer" containerID="645b402d2adf84d098fa27b8e9fe627ead642cf16058c5ce614ce1dfa5e81400"
Sep 17 23:32:51.801549 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:32:51.801525    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645b402d2adf84d098fa27b8e9fe627ead642cf16058c5ce614ce1dfa5e81400\": container with ID starting with 645b402d2adf84d098fa27b8e9fe627ead642cf16058c5ce614ce1dfa5e81400 not found: ID does not exist" containerID="645b402d2adf84d098fa27b8e9fe627ead642cf16058c5ce614ce1dfa5e81400"
Sep 17 23:32:51.801641 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.801557    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645b402d2adf84d098fa27b8e9fe627ead642cf16058c5ce614ce1dfa5e81400"} err="failed to get container status \"645b402d2adf84d098fa27b8e9fe627ead642cf16058c5ce614ce1dfa5e81400\": rpc error: code = NotFound desc = could not find container \"645b402d2adf84d098fa27b8e9fe627ead642cf16058c5ce614ce1dfa5e81400\": container with ID starting with 645b402d2adf84d098fa27b8e9fe627ead642cf16058c5ce614ce1dfa5e81400 not found: ID does not exist"
Sep 17 23:32:51.801641 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.801579    2568 scope.go:117] "RemoveContainer" containerID="b19c93b09400d67014dba5e6fdbc2c513d1afd6ab4b5c764748e5a6697761863"
Sep 17 23:32:51.812156 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.812138    2568 scope.go:117] "RemoveContainer" containerID="b19c93b09400d67014dba5e6fdbc2c513d1afd6ab4b5c764748e5a6697761863"
Sep 17 23:32:51.812397 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:32:51.812380    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19c93b09400d67014dba5e6fdbc2c513d1afd6ab4b5c764748e5a6697761863\": container with ID starting with b19c93b09400d67014dba5e6fdbc2c513d1afd6ab4b5c764748e5a6697761863 not found: ID does not exist" containerID="b19c93b09400d67014dba5e6fdbc2c513d1afd6ab4b5c764748e5a6697761863"
Sep 17 23:32:51.812447 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.812413    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19c93b09400d67014dba5e6fdbc2c513d1afd6ab4b5c764748e5a6697761863"} err="failed to get container status \"b19c93b09400d67014dba5e6fdbc2c513d1afd6ab4b5c764748e5a6697761863\": rpc error: code = NotFound desc = could not find container \"b19c93b09400d67014dba5e6fdbc2c513d1afd6ab4b5c764748e5a6697761863\": container with ID starting with b19c93b09400d67014dba5e6fdbc2c513d1afd6ab4b5c764748e5a6697761863 not found: ID does not exist"
Sep 17 23:32:51.812447 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.812427    2568 scope.go:117] "RemoveContainer" containerID="18d4d9bfc6b3e3b517501c169e0dda4e30b0fa310d66d906d0f3d4b0824c5d40"
Sep 17 23:32:51.820040 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.820022    2568 scope.go:117] "RemoveContainer" containerID="18d4d9bfc6b3e3b517501c169e0dda4e30b0fa310d66d906d0f3d4b0824c5d40"
Sep 17 23:32:51.820321 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:32:51.820302    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d4d9bfc6b3e3b517501c169e0dda4e30b0fa310d66d906d0f3d4b0824c5d40\": container with ID starting with 18d4d9bfc6b3e3b517501c169e0dda4e30b0fa310d66d906d0f3d4b0824c5d40 not found: ID does not exist" containerID="18d4d9bfc6b3e3b517501c169e0dda4e30b0fa310d66d906d0f3d4b0824c5d40"
Sep 17 23:32:51.820402 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.820330    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d4d9bfc6b3e3b517501c169e0dda4e30b0fa310d66d906d0f3d4b0824c5d40"} err="failed to get container status \"18d4d9bfc6b3e3b517501c169e0dda4e30b0fa310d66d906d0f3d4b0824c5d40\": rpc error: code = NotFound desc = could not find container \"18d4d9bfc6b3e3b517501c169e0dda4e30b0fa310d66d906d0f3d4b0824c5d40\": container with ID starting with 18d4d9bfc6b3e3b517501c169e0dda4e30b0fa310d66d906d0f3d4b0824c5d40 not found: ID does not exist"
Sep 17 23:32:51.820402 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.820352    2568 scope.go:117] "RemoveContainer" containerID="e9cc4f35b7abc80684b230b00f184c9e00d8317b2f9bd957c001c6b301141782"
Sep 17 23:32:51.821049 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.821025    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"]
Sep 17 23:32:51.823459 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.823440    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/distinct-w4-825f30-kserve-5cdc7d4b85-rhmxn"]
Sep 17 23:32:51.828230 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.828215    2568 scope.go:117] "RemoveContainer" containerID="e9cc4f35b7abc80684b230b00f184c9e00d8317b2f9bd957c001c6b301141782"
Sep 17 23:32:51.828451 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:32:51.828436    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9cc4f35b7abc80684b230b00f184c9e00d8317b2f9bd957c001c6b301141782\": container with ID starting with e9cc4f35b7abc80684b230b00f184c9e00d8317b2f9bd957c001c6b301141782 not found: ID does not exist" containerID="e9cc4f35b7abc80684b230b00f184c9e00d8317b2f9bd957c001c6b301141782"
Sep 17 23:32:51.828505 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.828456    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9cc4f35b7abc80684b230b00f184c9e00d8317b2f9bd957c001c6b301141782"} err="failed to get container status \"e9cc4f35b7abc80684b230b00f184c9e00d8317b2f9bd957c001c6b301141782\": rpc error: code = NotFound desc = could not find container \"e9cc4f35b7abc80684b230b00f184c9e00d8317b2f9bd957c001c6b301141782\": container with ID starting with e9cc4f35b7abc80684b230b00f184c9e00d8317b2f9bd957c001c6b301141782 not found: ID does not exist"
Sep 17 23:32:51.828505 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.828470    2568 scope.go:117] "RemoveContainer" containerID="9faf8e02d890277582e13db10e4df0c8682ca49b8aa6a86332fdb99657afafd5"
Sep 17 23:32:51.834958 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.834936    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"]
Sep 17 23:32:51.835786 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.835774    2568 scope.go:117] "RemoveContainer" containerID="9faf8e02d890277582e13db10e4df0c8682ca49b8aa6a86332fdb99657afafd5"
Sep 17 23:32:51.836061 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:32:51.836042    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9faf8e02d890277582e13db10e4df0c8682ca49b8aa6a86332fdb99657afafd5\": container with ID starting with 9faf8e02d890277582e13db10e4df0c8682ca49b8aa6a86332fdb99657afafd5 not found: ID does not exist" containerID="9faf8e02d890277582e13db10e4df0c8682ca49b8aa6a86332fdb99657afafd5"
Sep 17 23:32:51.836129 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.836065    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9faf8e02d890277582e13db10e4df0c8682ca49b8aa6a86332fdb99657afafd5"} err="failed to get container status \"9faf8e02d890277582e13db10e4df0c8682ca49b8aa6a86332fdb99657afafd5\": rpc error: code = NotFound desc = could not find container \"9faf8e02d890277582e13db10e4df0c8682ca49b8aa6a86332fdb99657afafd5\": container with ID starting with 9faf8e02d890277582e13db10e4df0c8682ca49b8aa6a86332fdb99657afafd5 not found: ID does not exist"
Sep 17 23:32:51.839006 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.838987    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-736809-5c578bf59-gg8sk"]
Sep 17 23:32:51.849665 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.849646    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"]
Sep 17 23:32:51.853671 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.853654    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/simulator-w4-825f30-kserve-648c59db6c-2kwzs"]
Sep 17 23:32:51.867493 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.867259    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"]
Sep 17 23:32:51.872370 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.872348    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/premium-w4-825f30-kserve-87fcb455d-54t52"]
Sep 17 23:32:51.888672 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.888646    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"]
Sep 17 23:32:51.890956 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.890935    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-825f30/distinct-2-w4-825f30-kserve-6d58bcf6c4-qvvc8"]
Sep 17 23:32:51.990319 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.990289    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="221ebd17-193c-4c22-9120-ebf7a4e4492e" path="/var/lib/kubelet/pods/221ebd17-193c-4c22-9120-ebf7a4e4492e/volumes"
Sep 17 23:32:51.990712 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.990698    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26631010-7003-4365-8935-004356931440" path="/var/lib/kubelet/pods/26631010-7003-4365-8935-004356931440/volumes"
Sep 17 23:32:51.991087 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.991073    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3028dec1-b28d-4853-a9fb-ab9578fbc4ba" path="/var/lib/kubelet/pods/3028dec1-b28d-4853-a9fb-ab9578fbc4ba/volumes"
Sep 17 23:32:51.991710 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.991696    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d35d4f5-288e-4f35-b646-d38e0c03ebf0" path="/var/lib/kubelet/pods/6d35d4f5-288e-4f35-b646-d38e0c03ebf0/volumes"
Sep 17 23:32:51.992095 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:51.992079    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a" path="/var/lib/kubelet/pods/7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a/volumes"
Sep 17 23:32:52.156325 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:52.156236    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:52.898512 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:52.898477    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:53.286960 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.286924    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj" podUID="29dcedab-f904-4ee6-ab58-ac399ab3de2b" containerName="main" containerID="cri-o://7ae44e29480b7102a969ab7c2d7ccf8e6be529ecdd014cd28c55ae536d8627f0" gracePeriod=45
Sep 17 23:32:53.303390 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.303355    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr" podUID="b7502b72-6269-4fe2-8a26-12018c6367e7" containerName="main" containerID="cri-o://c51eaefa9066b142d195855f192c8f52adb7e6b6af677f57653e2ef7e6f433a9" gracePeriod=45
Sep 17 23:32:53.554747 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.554724    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:32:53.575567 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.575544    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:32:53.725756 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.725722    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnlvn\" (UniqueName: \"kubernetes.io/projected/29dcedab-f904-4ee6-ab58-ac399ab3de2b-kube-api-access-jnlvn\") pod \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") "
Sep 17 23:32:53.725936 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.725769    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-home\") pod \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") "
Sep 17 23:32:53.725936 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.725797    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-tmp-dir\") pod \"b7502b72-6269-4fe2-8a26-12018c6367e7\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") "
Sep 17 23:32:53.725936 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.725830    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-home\") pod \"b7502b72-6269-4fe2-8a26-12018c6367e7\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") "
Sep 17 23:32:53.725936 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.725910    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7htjc\" (UniqueName: \"kubernetes.io/projected/b7502b72-6269-4fe2-8a26-12018c6367e7-kube-api-access-7htjc\") pod \"b7502b72-6269-4fe2-8a26-12018c6367e7\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") "
Sep 17 23:32:53.726101 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.725972    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29dcedab-f904-4ee6-ab58-ac399ab3de2b-tls-certs\") pod \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") "
Sep 17 23:32:53.726101 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726024    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-model-cache\") pod \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") "
Sep 17 23:32:53.726101 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726064    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-dshm\") pod \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") "
Sep 17 23:32:53.726101 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726061    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7502b72-6269-4fe2-8a26-12018c6367e7" (UID: "b7502b72-6269-4fe2-8a26-12018c6367e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:53.726304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726105    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-home" (OuterVolumeSpecName: "home") pod "29dcedab-f904-4ee6-ab58-ac399ab3de2b" (UID: "29dcedab-f904-4ee6-ab58-ac399ab3de2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:53.726304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726110    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7502b72-6269-4fe2-8a26-12018c6367e7-tls-certs\") pod \"b7502b72-6269-4fe2-8a26-12018c6367e7\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") "
Sep 17 23:32:53.726304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726155    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-home" (OuterVolumeSpecName: "home") pod "b7502b72-6269-4fe2-8a26-12018c6367e7" (UID: "b7502b72-6269-4fe2-8a26-12018c6367e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:53.726304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726179    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-dshm\") pod \"b7502b72-6269-4fe2-8a26-12018c6367e7\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") "
Sep 17 23:32:53.726304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726218    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-tmp-dir\") pod \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\" (UID: \"29dcedab-f904-4ee6-ab58-ac399ab3de2b\") "
Sep 17 23:32:53.726304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726233    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-model-cache" (OuterVolumeSpecName: "model-cache") pod "29dcedab-f904-4ee6-ab58-ac399ab3de2b" (UID: "29dcedab-f904-4ee6-ab58-ac399ab3de2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:53.726304 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726250    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-model-cache\") pod \"b7502b72-6269-4fe2-8a26-12018c6367e7\" (UID: \"b7502b72-6269-4fe2-8a26-12018c6367e7\") "
Sep 17 23:32:53.726684 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726602    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29dcedab-f904-4ee6-ab58-ac399ab3de2b" (UID: "29dcedab-f904-4ee6-ab58-ac399ab3de2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:53.726684 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726628    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-model-cache\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:53.726684 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726651    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-home\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:53.726847 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726694    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-tmp-dir\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:53.726847 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726707    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-home\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:53.726847 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.726811    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7502b72-6269-4fe2-8a26-12018c6367e7" (UID: "b7502b72-6269-4fe2-8a26-12018c6367e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:53.728365 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.728334    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29dcedab-f904-4ee6-ab58-ac399ab3de2b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29dcedab-f904-4ee6-ab58-ac399ab3de2b" (UID: "29dcedab-f904-4ee6-ab58-ac399ab3de2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:53.728470 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.728387    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-dshm" (OuterVolumeSpecName: "dshm") pod "b7502b72-6269-4fe2-8a26-12018c6367e7" (UID: "b7502b72-6269-4fe2-8a26-12018c6367e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:53.728470 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.728431    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7502b72-6269-4fe2-8a26-12018c6367e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7502b72-6269-4fe2-8a26-12018c6367e7" (UID: "b7502b72-6269-4fe2-8a26-12018c6367e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:53.728578 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.728481    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29dcedab-f904-4ee6-ab58-ac399ab3de2b-kube-api-access-jnlvn" (OuterVolumeSpecName: "kube-api-access-jnlvn") pod "29dcedab-f904-4ee6-ab58-ac399ab3de2b" (UID: "29dcedab-f904-4ee6-ab58-ac399ab3de2b"). InnerVolumeSpecName "kube-api-access-jnlvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:53.728578 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.728516    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-dshm" (OuterVolumeSpecName: "dshm") pod "29dcedab-f904-4ee6-ab58-ac399ab3de2b" (UID: "29dcedab-f904-4ee6-ab58-ac399ab3de2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:53.728578 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.728549    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7502b72-6269-4fe2-8a26-12018c6367e7-kube-api-access-7htjc" (OuterVolumeSpecName: "kube-api-access-7htjc") pod "b7502b72-6269-4fe2-8a26-12018c6367e7" (UID: "b7502b72-6269-4fe2-8a26-12018c6367e7"). InnerVolumeSpecName "kube-api-access-7htjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:53.812199 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.812121    2568 generic.go:356] "Generic (PLEG): container finished" podID="b7502b72-6269-4fe2-8a26-12018c6367e7" containerID="c51eaefa9066b142d195855f192c8f52adb7e6b6af677f57653e2ef7e6f433a9" exitCode=0
Sep 17 23:32:53.812199 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.812167    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"
Sep 17 23:32:53.812429 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.812202    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr" event={"ID":"b7502b72-6269-4fe2-8a26-12018c6367e7","Type":"ContainerDied","Data":"c51eaefa9066b142d195855f192c8f52adb7e6b6af677f57653e2ef7e6f433a9"}
Sep 17 23:32:53.812429 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.812250    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr" event={"ID":"b7502b72-6269-4fe2-8a26-12018c6367e7","Type":"ContainerDied","Data":"144d5911257b50907ad3e6edcf4304754afc0d46b35f3931479e99c759b409ab"}
Sep 17 23:32:53.812429 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.812272    2568 scope.go:117] "RemoveContainer" containerID="c51eaefa9066b142d195855f192c8f52adb7e6b6af677f57653e2ef7e6f433a9"
Sep 17 23:32:53.813611 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.813588    2568 generic.go:356] "Generic (PLEG): container finished" podID="29dcedab-f904-4ee6-ab58-ac399ab3de2b" containerID="7ae44e29480b7102a969ab7c2d7ccf8e6be529ecdd014cd28c55ae536d8627f0" exitCode=0
Sep 17 23:32:53.813760 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.813625    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"
Sep 17 23:32:53.813760 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.813682    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj" event={"ID":"29dcedab-f904-4ee6-ab58-ac399ab3de2b","Type":"ContainerDied","Data":"7ae44e29480b7102a969ab7c2d7ccf8e6be529ecdd014cd28c55ae536d8627f0"}
Sep 17 23:32:53.813760 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.813718    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj" event={"ID":"29dcedab-f904-4ee6-ab58-ac399ab3de2b","Type":"ContainerDied","Data":"3a2cb4ebda3d187ab352ac7971969376a91384daf7895de5af167b68c35e5634"}
Sep 17 23:32:53.821687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.821663    2568 scope.go:117] "RemoveContainer" containerID="c51eaefa9066b142d195855f192c8f52adb7e6b6af677f57653e2ef7e6f433a9"
Sep 17 23:32:53.822003 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:32:53.821980    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51eaefa9066b142d195855f192c8f52adb7e6b6af677f57653e2ef7e6f433a9\": container with ID starting with c51eaefa9066b142d195855f192c8f52adb7e6b6af677f57653e2ef7e6f433a9 not found: ID does not exist" containerID="c51eaefa9066b142d195855f192c8f52adb7e6b6af677f57653e2ef7e6f433a9"
Sep 17 23:32:53.822086 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.822013    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51eaefa9066b142d195855f192c8f52adb7e6b6af677f57653e2ef7e6f433a9"} err="failed to get container status \"c51eaefa9066b142d195855f192c8f52adb7e6b6af677f57653e2ef7e6f433a9\": rpc error: code = NotFound desc = could not find container \"c51eaefa9066b142d195855f192c8f52adb7e6b6af677f57653e2ef7e6f433a9\": container with ID starting with c51eaefa9066b142d195855f192c8f52adb7e6b6af677f57653e2ef7e6f433a9 not found: ID does not exist"
Sep 17 23:32:53.822086 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.822037    2568 scope.go:117] "RemoveContainer" containerID="7ae44e29480b7102a969ab7c2d7ccf8e6be529ecdd014cd28c55ae536d8627f0"
Sep 17 23:32:53.827680 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.827656    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7htjc\" (UniqueName: \"kubernetes.io/projected/b7502b72-6269-4fe2-8a26-12018c6367e7-kube-api-access-7htjc\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:53.827768 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.827686    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29dcedab-f904-4ee6-ab58-ac399ab3de2b-tls-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:53.827768 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.827700    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-dshm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:53.827768 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.827712    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7502b72-6269-4fe2-8a26-12018c6367e7-tls-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:53.827768 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.827724    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-dshm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:53.827768 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.827736    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29dcedab-f904-4ee6-ab58-ac399ab3de2b-tmp-dir\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:53.827768 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.827747    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7502b72-6269-4fe2-8a26-12018c6367e7-model-cache\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:53.827768 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.827759    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnlvn\" (UniqueName: \"kubernetes.io/projected/29dcedab-f904-4ee6-ab58-ac399ab3de2b-kube-api-access-jnlvn\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:32:53.829933 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.829909    2568 scope.go:117] "RemoveContainer" containerID="7ae44e29480b7102a969ab7c2d7ccf8e6be529ecdd014cd28c55ae536d8627f0"
Sep 17 23:32:53.830166 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:32:53.830150    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae44e29480b7102a969ab7c2d7ccf8e6be529ecdd014cd28c55ae536d8627f0\": container with ID starting with 7ae44e29480b7102a969ab7c2d7ccf8e6be529ecdd014cd28c55ae536d8627f0 not found: ID does not exist" containerID="7ae44e29480b7102a969ab7c2d7ccf8e6be529ecdd014cd28c55ae536d8627f0"
Sep 17 23:32:53.830208 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.830171    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae44e29480b7102a969ab7c2d7ccf8e6be529ecdd014cd28c55ae536d8627f0"} err="failed to get container status \"7ae44e29480b7102a969ab7c2d7ccf8e6be529ecdd014cd28c55ae536d8627f0\": rpc error: code = NotFound desc = could not find container \"7ae44e29480b7102a969ab7c2d7ccf8e6be529ecdd014cd28c55ae536d8627f0\": container with ID starting with 7ae44e29480b7102a969ab7c2d7ccf8e6be529ecdd014cd28c55ae536d8627f0 not found: ID does not exist"
Sep 17 23:32:53.837606 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.837584    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"]
Sep 17 23:32:53.840905 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.840885    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-736809/distinct-2-w0-736809-kserve-755f8dfbc9-bw7nj"]
Sep 17 23:32:53.851575 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.851550    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"]
Sep 17 23:32:53.857257 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.857237    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-736809/embedding-w0-736809-kserve-65996558d9-plcvr"]
Sep 17 23:32:53.990845 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.990814    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29dcedab-f904-4ee6-ab58-ac399ab3de2b" path="/var/lib/kubelet/pods/29dcedab-f904-4ee6-ab58-ac399ab3de2b/volumes"
Sep 17 23:32:53.991252 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:53.991238    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7502b72-6269-4fe2-8a26-12018c6367e7" path="/var/lib/kubelet/pods/b7502b72-6269-4fe2-8a26-12018c6367e7/volumes"
Sep 17 23:32:56.879778 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:56.879747    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:56.979450 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:56.979421    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:57.142630 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:32:57.142558    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:12.052423 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:12.052383    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:12.152084 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:12.152043    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:13.301378 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:13.301315    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:15.849192 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:15.849158    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:15.902548 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:15.902509    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:16.014910 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:16.014879    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:18.761112 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:18.761061    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:20.049600 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:20.049553    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:20.296422 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:20.296381    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:21.630269 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:21.630223    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:21.977251 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:21.977173    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"]
Sep 17 23:33:21.998974 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:21.998938    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:27.400213 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:27.400181    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:36.999758 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:36.999695    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg" podUID="b8401988-f3c3-4357-8185-98fd7cab6dda" containerName="main" containerID="cri-o://a1fed6c4624c9d323340b3c904a1400f775463d82dc3bb62345e4b7c409975a9" gracePeriod=45
Sep 17 23:33:37.250222 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.250159    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:33:37.435225 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.435174    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-tmp-dir\") pod \"b8401988-f3c3-4357-8185-98fd7cab6dda\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") "
Sep 17 23:33:37.435448 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.435259    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-dshm\") pod \"b8401988-f3c3-4357-8185-98fd7cab6dda\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") "
Sep 17 23:33:37.435448 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.435298    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-home\") pod \"b8401988-f3c3-4357-8185-98fd7cab6dda\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") "
Sep 17 23:33:37.435448 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.435333    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8401988-f3c3-4357-8185-98fd7cab6dda-tls-certs\") pod \"b8401988-f3c3-4357-8185-98fd7cab6dda\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") "
Sep 17 23:33:37.435448 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.435423    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-model-cache\") pod \"b8401988-f3c3-4357-8185-98fd7cab6dda\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") "
Sep 17 23:33:37.435679 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.435463    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbn87\" (UniqueName: \"kubernetes.io/projected/b8401988-f3c3-4357-8185-98fd7cab6dda-kube-api-access-dbn87\") pod \"b8401988-f3c3-4357-8185-98fd7cab6dda\" (UID: \"b8401988-f3c3-4357-8185-98fd7cab6dda\") "
Sep 17 23:33:37.435679 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.435492    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b8401988-f3c3-4357-8185-98fd7cab6dda" (UID: "b8401988-f3c3-4357-8185-98fd7cab6dda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:37.435679 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.435582    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-home" (OuterVolumeSpecName: "home") pod "b8401988-f3c3-4357-8185-98fd7cab6dda" (UID: "b8401988-f3c3-4357-8185-98fd7cab6dda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:37.435796 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.435730    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-model-cache" (OuterVolumeSpecName: "model-cache") pod "b8401988-f3c3-4357-8185-98fd7cab6dda" (UID: "b8401988-f3c3-4357-8185-98fd7cab6dda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:37.435834 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.435818    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-model-cache\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:33:37.435890 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.435837    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-tmp-dir\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:33:37.435890 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.435849    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-home\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:33:37.437688 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.437662    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8401988-f3c3-4357-8185-98fd7cab6dda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8401988-f3c3-4357-8185-98fd7cab6dda" (UID: "b8401988-f3c3-4357-8185-98fd7cab6dda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:37.437795 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.437697    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-dshm" (OuterVolumeSpecName: "dshm") pod "b8401988-f3c3-4357-8185-98fd7cab6dda" (UID: "b8401988-f3c3-4357-8185-98fd7cab6dda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:37.437795 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.437706    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8401988-f3c3-4357-8185-98fd7cab6dda-kube-api-access-dbn87" (OuterVolumeSpecName: "kube-api-access-dbn87") pod "b8401988-f3c3-4357-8185-98fd7cab6dda" (UID: "b8401988-f3c3-4357-8185-98fd7cab6dda"). InnerVolumeSpecName "kube-api-access-dbn87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:37.537253 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.537219    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbn87\" (UniqueName: \"kubernetes.io/projected/b8401988-f3c3-4357-8185-98fd7cab6dda-kube-api-access-dbn87\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:33:37.537253 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.537252    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8401988-f3c3-4357-8185-98fd7cab6dda-dshm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:33:37.537461 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:37.537262    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8401988-f3c3-4357-8185-98fd7cab6dda-tls-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:33:38.041709 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:38.041673    2568 generic.go:356] "Generic (PLEG): container finished" podID="b8401988-f3c3-4357-8185-98fd7cab6dda" containerID="a1fed6c4624c9d323340b3c904a1400f775463d82dc3bb62345e4b7c409975a9" exitCode=0
Sep 17 23:33:38.042214 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:38.041719    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"
Sep 17 23:33:38.042214 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:38.041742    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg" event={"ID":"b8401988-f3c3-4357-8185-98fd7cab6dda","Type":"ContainerDied","Data":"a1fed6c4624c9d323340b3c904a1400f775463d82dc3bb62345e4b7c409975a9"}
Sep 17 23:33:38.042214 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:38.041774    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg" event={"ID":"b8401988-f3c3-4357-8185-98fd7cab6dda","Type":"ContainerDied","Data":"d9522837f7ed13253eeeffb31467ac1aaf6aa31bb18608507c67803931c45a6b"}
Sep 17 23:33:38.042214 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:38.041790    2568 scope.go:117] "RemoveContainer" containerID="a1fed6c4624c9d323340b3c904a1400f775463d82dc3bb62345e4b7c409975a9"
Sep 17 23:33:38.050168 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:38.050147    2568 scope.go:117] "RemoveContainer" containerID="a1fed6c4624c9d323340b3c904a1400f775463d82dc3bb62345e4b7c409975a9"
Sep 17 23:33:38.050400 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:33:38.050382    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fed6c4624c9d323340b3c904a1400f775463d82dc3bb62345e4b7c409975a9\": container with ID starting with a1fed6c4624c9d323340b3c904a1400f775463d82dc3bb62345e4b7c409975a9 not found: ID does not exist" containerID="a1fed6c4624c9d323340b3c904a1400f775463d82dc3bb62345e4b7c409975a9"
Sep 17 23:33:38.050442 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:38.050414    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fed6c4624c9d323340b3c904a1400f775463d82dc3bb62345e4b7c409975a9"} err="failed to get container status \"a1fed6c4624c9d323340b3c904a1400f775463d82dc3bb62345e4b7c409975a9\": rpc error: code = NotFound desc = could not find container \"a1fed6c4624c9d323340b3c904a1400f775463d82dc3bb62345e4b7c409975a9\": container with ID starting with a1fed6c4624c9d323340b3c904a1400f775463d82dc3bb62345e4b7c409975a9 not found: ID does not exist"
Sep 17 23:33:38.057639 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:38.057614    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"]
Sep 17 23:33:38.062959 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:38.062938    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6ac6e/auth-test-model-f6ac6e-kserve-5f759c9695-vs2mg"]
Sep 17 23:33:39.990019 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:39.989982    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8401988-f3c3-4357-8185-98fd7cab6dda" path="/var/lib/kubelet/pods/b8401988-f3c3-4357-8185-98fd7cab6dda/volumes"
Sep 17 23:33:43.126364 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:33:43.126336    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:11.022536 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:11.022500    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:11.123163 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:11.123131    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:13.803507 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803467    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"]
Sep 17 23:34:13.803891 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803882    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8401988-f3c3-4357-8185-98fd7cab6dda" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803895    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8401988-f3c3-4357-8185-98fd7cab6dda" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803906    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221ebd17-193c-4c22-9120-ebf7a4e4492e" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803912    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="221ebd17-193c-4c22-9120-ebf7a4e4492e" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803922    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26631010-7003-4365-8935-004356931440" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803928    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="26631010-7003-4365-8935-004356931440" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803933    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d3af52-3619-488c-bd29-65084d670614" containerName="maas-api"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803939    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d3af52-3619-488c-bd29-65084d670614" containerName="maas-api"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803948    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c2012bd-9df1-48e9-9c40-17e7e4f43eb0" containerName="maas-api"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803953    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c2012bd-9df1-48e9-9c40-17e7e4f43eb0" containerName="maas-api"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803961    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7502b72-6269-4fe2-8a26-12018c6367e7" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803966    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7502b72-6269-4fe2-8a26-12018c6367e7" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803972    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a" containerName="maas-api"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803977    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a" containerName="maas-api"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803988    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71f11e1b-28fb-4b8b-b6af-198c5decffd4" containerName="maas-api"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.803994    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="71f11e1b-28fb-4b8b-b6af-198c5decffd4" containerName="maas-api"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804002    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d35d4f5-288e-4f35-b646-d38e0c03ebf0" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804007    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d35d4f5-288e-4f35-b646-d38e0c03ebf0" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804012    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3028dec1-b28d-4853-a9fb-ab9578fbc4ba" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804017    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3028dec1-b28d-4853-a9fb-ab9578fbc4ba" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804022    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29dcedab-f904-4ee6-ab58-ac399ab3de2b" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804027    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="29dcedab-f904-4ee6-ab58-ac399ab3de2b" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804077    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="29dcedab-f904-4ee6-ab58-ac399ab3de2b" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804085    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7502b72-6269-4fe2-8a26-12018c6367e7" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804092    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8401988-f3c3-4357-8185-98fd7cab6dda" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804097    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c2012bd-9df1-48e9-9c40-17e7e4f43eb0" containerName="maas-api"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804102    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="26631010-7003-4365-8935-004356931440" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804109    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3028dec1-b28d-4853-a9fb-ab9578fbc4ba" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804115    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d35d4f5-288e-4f35-b646-d38e0c03ebf0" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804120    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa69cff-a7ca-4d9a-ae45-9a9a1fc8e08a" containerName="maas-api"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804126    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="71f11e1b-28fb-4b8b-b6af-198c5decffd4" containerName="maas-api"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804132    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="221ebd17-193c-4c22-9120-ebf7a4e4492e" containerName="main"
Sep 17 23:34:13.804172 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.804137    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d3af52-3619-488c-bd29-65084d670614" containerName="maas-api"
Sep 17 23:34:13.807319 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.807302    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:13.809855 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.809829    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6ac6e\"/\"kube-root-ca.crt\""
Sep 17 23:34:13.810785 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.810756    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6ac6e\"/\"openshift-service-ca.crt\""
Sep 17 23:34:13.810919 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.810803    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6ac6e\"/\"default-dockercfg-5cvdt\""
Sep 17 23:34:13.810919 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.810796    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f6ac6e\"/\"sub-test-model-f6ac6e-kserve-self-signed-certs\""
Sep 17 23:34:13.817537 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.817516    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"]
Sep 17 23:34:13.834431 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.834396    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:13.986707 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.986672    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-dshm\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:13.986707 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.986707    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-tmp-dir\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:13.986945 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.986749    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f8f664-61f7-42be-95a3-743158705489-tls-certs\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:13.986945 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.986813    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-home\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:13.986945 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.986890    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdjfm\" (UniqueName: \"kubernetes.io/projected/27f8f664-61f7-42be-95a3-743158705489-kube-api-access-vdjfm\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:13.986945 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:13.986917    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-model-cache\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:14.088044 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:14.087964    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-dshm\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:14.088044 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:14.088001    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-tmp-dir\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:14.088044 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:14.088036    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f8f664-61f7-42be-95a3-743158705489-tls-certs\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:14.088328 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:14.088067    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-home\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:14.088328 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:14.088138    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdjfm\" (UniqueName: \"kubernetes.io/projected/27f8f664-61f7-42be-95a3-743158705489-kube-api-access-vdjfm\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:14.088328 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:14.088186    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-model-cache\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:14.088497 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:14.088438    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-home\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:14.088497 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:14.088481    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-tmp-dir\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:14.088631 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:14.088611    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-model-cache\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:14.090119 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:14.090102    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-dshm\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:14.090421 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:14.090405    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f8f664-61f7-42be-95a3-743158705489-tls-certs\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:14.095454 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:14.095433    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdjfm\" (UniqueName: \"kubernetes.io/projected/27f8f664-61f7-42be-95a3-743158705489-kube-api-access-vdjfm\") pod \"sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") " pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:14.117446 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:14.117429    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:14.386782 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:14.386760    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"]
Sep 17 23:34:14.388961 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:34:14.388927    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27f8f664_61f7_42be_95a3_743158705489.slice/crio-3b3c902790f59814e465cc6eef780112610eea2a9550a219e417b208737f9d62 WatchSource:0}: Error finding container 3b3c902790f59814e465cc6eef780112610eea2a9550a219e417b208737f9d62: Status 404 returned error can't find the container with id 3b3c902790f59814e465cc6eef780112610eea2a9550a219e417b208737f9d62
Sep 17 23:34:15.228945 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:15.228909    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx" event={"ID":"27f8f664-61f7-42be-95a3-743158705489","Type":"ContainerStarted","Data":"9aba878b2f24add01d9a43d882a14a98e9a772242db1bb961e0ec13f906dd02b"}
Sep 17 23:34:15.229340 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:15.228954    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx" event={"ID":"27f8f664-61f7-42be-95a3-743158705489","Type":"ContainerStarted","Data":"3b3c902790f59814e465cc6eef780112610eea2a9550a219e417b208737f9d62"}
Sep 17 23:34:15.245397 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:15.245346    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx" podStartSLOduration=2.245332187 podStartE2EDuration="2.245332187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:34:15.243645731 +0000 UTC m=+1653.467321472" watchObservedRunningTime="2026-09-17 23:34:15.245332187 +0000 UTC m=+1653.469007996"
Sep 17 23:34:24.117653 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:24.117620    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:24.118180 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:24.117675    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:24.130128 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:24.130103    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:24.288403 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:24.288376    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:34:25.008001 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:25.007961    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:25.108577 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:25.108547    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:46.342439 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:46.342362    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:47.292445 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:47.292410    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"]
Sep 17 23:34:47.312335 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:47.312292    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:56.096692 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.096616    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"]
Sep 17 23:34:56.100168 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.100148    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.103505 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.103472    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f80c49\"/\"default-dockercfg-hhpdp\""
Sep 17 23:34:56.103616 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.103579    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f80c49\"/\"rate-test-model-f80c49-kserve-self-signed-certs\""
Sep 17 23:34:56.103616 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.103603    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f80c49\"/\"kube-root-ca.crt\""
Sep 17 23:34:56.103731 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.103589    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f80c49\"/\"openshift-service-ca.crt\""
Sep 17 23:34:56.112958 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.112929    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"]
Sep 17 23:34:56.135252 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.135227    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:56.278723 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.278684    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-model-cache\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.278917 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.278730    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fef4ca-c90a-41eb-88a0-17178af95977-tls-certs\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.278917 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.278753    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-home\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.278917 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.278850    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-dshm\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.278917 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.278899    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-tmp-dir\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.279072 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.278952    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5bbj\" (UniqueName: \"kubernetes.io/projected/e3fef4ca-c90a-41eb-88a0-17178af95977-kube-api-access-w5bbj\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.379761 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.379656    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-model-cache\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.379761 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.379706    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fef4ca-c90a-41eb-88a0-17178af95977-tls-certs\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.379761 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.379748    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-home\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.380092 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.379809    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-dshm\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.380092 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.379832    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-tmp-dir\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.380092 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.379861    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5bbj\" (UniqueName: \"kubernetes.io/projected/e3fef4ca-c90a-41eb-88a0-17178af95977-kube-api-access-w5bbj\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.380239 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.380104    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-model-cache\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.380239 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.380129    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-home\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.380320 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.380267    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-tmp-dir\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.382080 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.382060    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-dshm\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.382405 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.382386    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fef4ca-c90a-41eb-88a0-17178af95977-tls-certs\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.387953 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.387929    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5bbj\" (UniqueName: \"kubernetes.io/projected/e3fef4ca-c90a-41eb-88a0-17178af95977-kube-api-access-w5bbj\") pod \"rate-test-model-f80c49-kserve-665fbc6887-mfmn8\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") " pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.408913 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.408890    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:34:56.679071 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:56.679046    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"]
Sep 17 23:34:56.681550 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:34:56.681510    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3fef4ca_c90a_41eb_88a0_17178af95977.slice/crio-8b074d307eb5225a5e98ea9e1c657586250d1efc7fdfa11d989dd77c47d65fc6 WatchSource:0}: Error finding container 8b074d307eb5225a5e98ea9e1c657586250d1efc7fdfa11d989dd77c47d65fc6: Status 404 returned error can't find the container with id 8b074d307eb5225a5e98ea9e1c657586250d1efc7fdfa11d989dd77c47d65fc6
Sep 17 23:34:57.439367 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:57.439326    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8" event={"ID":"e3fef4ca-c90a-41eb-88a0-17178af95977","Type":"ContainerStarted","Data":"05b09ca3c71780a1cb0fbf64a3826173f5c0b56d2e759724150eab31dca17ffb"}
Sep 17 23:34:57.439367 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:57.439367    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8" event={"ID":"e3fef4ca-c90a-41eb-88a0-17178af95977","Type":"ContainerStarted","Data":"8b074d307eb5225a5e98ea9e1c657586250d1efc7fdfa11d989dd77c47d65fc6"}
Sep 17 23:34:57.457092 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:34:57.457041    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8" podStartSLOduration=1.4570246230000001 podStartE2EDuration="1.457024623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:34:57.454446195 +0000 UTC m=+1695.678121942" watchObservedRunningTime="2026-09-17 23:34:57.457024623 +0000 UTC m=+1695.680700415"
Sep 17 23:35:02.311610 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.311571    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx" podUID="27f8f664-61f7-42be-95a3-743158705489" containerName="main" containerID="cri-o://9aba878b2f24add01d9a43d882a14a98e9a772242db1bb961e0ec13f906dd02b" gracePeriod=45
Sep 17 23:35:02.463967 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.463919    2568 generic.go:356] "Generic (PLEG): container finished" podID="27f8f664-61f7-42be-95a3-743158705489" containerID="9aba878b2f24add01d9a43d882a14a98e9a772242db1bb961e0ec13f906dd02b" exitCode=0
Sep 17 23:35:02.463967 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.463969    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx" event={"ID":"27f8f664-61f7-42be-95a3-743158705489","Type":"ContainerDied","Data":"9aba878b2f24add01d9a43d882a14a98e9a772242db1bb961e0ec13f906dd02b"}
Sep 17 23:35:02.551042 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.551018    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:35:02.737330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.737245    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f8f664-61f7-42be-95a3-743158705489-tls-certs\") pod \"27f8f664-61f7-42be-95a3-743158705489\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") "
Sep 17 23:35:02.737330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.737320    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-home\") pod \"27f8f664-61f7-42be-95a3-743158705489\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") "
Sep 17 23:35:02.737564 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.737340    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdjfm\" (UniqueName: \"kubernetes.io/projected/27f8f664-61f7-42be-95a3-743158705489-kube-api-access-vdjfm\") pod \"27f8f664-61f7-42be-95a3-743158705489\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") "
Sep 17 23:35:02.737564 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.737357    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-tmp-dir\") pod \"27f8f664-61f7-42be-95a3-743158705489\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") "
Sep 17 23:35:02.737564 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.737399    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-model-cache\") pod \"27f8f664-61f7-42be-95a3-743158705489\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") "
Sep 17 23:35:02.737564 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.737429    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-dshm\") pod \"27f8f664-61f7-42be-95a3-743158705489\" (UID: \"27f8f664-61f7-42be-95a3-743158705489\") "
Sep 17 23:35:02.737764 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.737592    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-home" (OuterVolumeSpecName: "home") pod "27f8f664-61f7-42be-95a3-743158705489" (UID: "27f8f664-61f7-42be-95a3-743158705489"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:02.737764 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.737680    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27f8f664-61f7-42be-95a3-743158705489" (UID: "27f8f664-61f7-42be-95a3-743158705489"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:02.737764 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.737687    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-home\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:35:02.737764 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.737710    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-model-cache" (OuterVolumeSpecName: "model-cache") pod "27f8f664-61f7-42be-95a3-743158705489" (UID: "27f8f664-61f7-42be-95a3-743158705489"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:02.739315 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.739286    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f8f664-61f7-42be-95a3-743158705489-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27f8f664-61f7-42be-95a3-743158705489" (UID: "27f8f664-61f7-42be-95a3-743158705489"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:35:02.739426 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.739385    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f8f664-61f7-42be-95a3-743158705489-kube-api-access-vdjfm" (OuterVolumeSpecName: "kube-api-access-vdjfm") pod "27f8f664-61f7-42be-95a3-743158705489" (UID: "27f8f664-61f7-42be-95a3-743158705489"). InnerVolumeSpecName "kube-api-access-vdjfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:35:02.739466 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.739450    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-dshm" (OuterVolumeSpecName: "dshm") pod "27f8f664-61f7-42be-95a3-743158705489" (UID: "27f8f664-61f7-42be-95a3-743158705489"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:02.838221 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.838190    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-dshm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:35:02.838221 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.838215    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f8f664-61f7-42be-95a3-743158705489-tls-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:35:02.838221 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.838225    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdjfm\" (UniqueName: \"kubernetes.io/projected/27f8f664-61f7-42be-95a3-743158705489-kube-api-access-vdjfm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:35:02.838457 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.838250    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-tmp-dir\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:35:02.838457 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:02.838260    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f8f664-61f7-42be-95a3-743158705489-model-cache\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:35:03.469651 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:03.469614    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx" event={"ID":"27f8f664-61f7-42be-95a3-743158705489","Type":"ContainerDied","Data":"3b3c902790f59814e465cc6eef780112610eea2a9550a219e417b208737f9d62"}
Sep 17 23:35:03.469651 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:03.469637    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"
Sep 17 23:35:03.470161 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:03.469668    2568 scope.go:117] "RemoveContainer" containerID="9aba878b2f24add01d9a43d882a14a98e9a772242db1bb961e0ec13f906dd02b"
Sep 17 23:35:03.490321 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:03.490291    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"]
Sep 17 23:35:03.493420 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:03.493398    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f6ac6e/sub-test-model-f6ac6e-kserve-6cb7f7bb85-l67xx"]
Sep 17 23:35:03.990169 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:03.990136    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27f8f664-61f7-42be-95a3-743158705489" path="/var/lib/kubelet/pods/27f8f664-61f7-42be-95a3-743158705489/volumes"
Sep 17 23:35:06.409892 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:06.409836    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:35:06.410336 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:06.409925    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:35:06.422798 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:06.422770    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:35:06.495470 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:06.495438    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:35:30.737882 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:30.737838    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:46.118195 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:46.118159    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:48.701729 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:48.701693    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:54.393230 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:54.393191    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:35:54.451360 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:35:54.451329    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:14.641847 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:14.641813    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"]
Sep 17 23:36:14.661114 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:14.661079    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:15.618100 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:15.618068    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:16.058528 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:16.058498    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:16.357476 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:16.357397    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:16.659711 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:16.659632    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:18.444166 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:18.444136    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:19.106988 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:19.106946    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:19.222831 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:19.222800    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:19.321389 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:19.321363    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:19.423662 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:19.423573    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:20.038368 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:20.038336    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:21.053420 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:21.053394    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:21.158727 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:21.158696    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:21.653191 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:21.653148    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:21.751885 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:21.751829    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:21.850018 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:21.849982    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:29.661127 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.661068    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8" podUID="e3fef4ca-c90a-41eb-88a0-17178af95977" containerName="main" containerID="cri-o://05b09ca3c71780a1cb0fbf64a3826173f5c0b56d2e759724150eab31dca17ffb" gracePeriod=45
Sep 17 23:36:29.885101 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.885063    2568 generic.go:356] "Generic (PLEG): container finished" podID="e3fef4ca-c90a-41eb-88a0-17178af95977" containerID="05b09ca3c71780a1cb0fbf64a3826173f5c0b56d2e759724150eab31dca17ffb" exitCode=0
Sep 17 23:36:29.885299 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.885114    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8" event={"ID":"e3fef4ca-c90a-41eb-88a0-17178af95977","Type":"ContainerDied","Data":"05b09ca3c71780a1cb0fbf64a3826173f5c0b56d2e759724150eab31dca17ffb"}
Sep 17 23:36:29.909257 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.909234    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:36:29.982816 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.982736    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-home\") pod \"e3fef4ca-c90a-41eb-88a0-17178af95977\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") "
Sep 17 23:36:29.982816 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.982789    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-model-cache\") pod \"e3fef4ca-c90a-41eb-88a0-17178af95977\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") "
Sep 17 23:36:29.982816 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.982817    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-dshm\") pod \"e3fef4ca-c90a-41eb-88a0-17178af95977\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") "
Sep 17 23:36:29.983102 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.982850    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-tmp-dir\") pod \"e3fef4ca-c90a-41eb-88a0-17178af95977\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") "
Sep 17 23:36:29.983102 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.982887    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fef4ca-c90a-41eb-88a0-17178af95977-tls-certs\") pod \"e3fef4ca-c90a-41eb-88a0-17178af95977\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") "
Sep 17 23:36:29.983102 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.982921    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5bbj\" (UniqueName: \"kubernetes.io/projected/e3fef4ca-c90a-41eb-88a0-17178af95977-kube-api-access-w5bbj\") pod \"e3fef4ca-c90a-41eb-88a0-17178af95977\" (UID: \"e3fef4ca-c90a-41eb-88a0-17178af95977\") "
Sep 17 23:36:29.983102 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.983067    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-home" (OuterVolumeSpecName: "home") pod "e3fef4ca-c90a-41eb-88a0-17178af95977" (UID: "e3fef4ca-c90a-41eb-88a0-17178af95977"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:36:29.983102 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.983074    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3fef4ca-c90a-41eb-88a0-17178af95977" (UID: "e3fef4ca-c90a-41eb-88a0-17178af95977"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:36:29.983339 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.983154    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3fef4ca-c90a-41eb-88a0-17178af95977" (UID: "e3fef4ca-c90a-41eb-88a0-17178af95977"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:36:29.983339 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.983222    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-model-cache\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:36:29.983339 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.983254    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-tmp-dir\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:36:29.983339 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.983263    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-home\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:36:29.985292 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.985260    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3fef4ca-c90a-41eb-88a0-17178af95977-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3fef4ca-c90a-41eb-88a0-17178af95977" (UID: "e3fef4ca-c90a-41eb-88a0-17178af95977"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:36:29.985292 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.985272    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3fef4ca-c90a-41eb-88a0-17178af95977-kube-api-access-w5bbj" (OuterVolumeSpecName: "kube-api-access-w5bbj") pod "e3fef4ca-c90a-41eb-88a0-17178af95977" (UID: "e3fef4ca-c90a-41eb-88a0-17178af95977"). InnerVolumeSpecName "kube-api-access-w5bbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:36:29.985447 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:29.985340    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-dshm" (OuterVolumeSpecName: "dshm") pod "e3fef4ca-c90a-41eb-88a0-17178af95977" (UID: "e3fef4ca-c90a-41eb-88a0-17178af95977"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:36:30.084164 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:30.084137    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3fef4ca-c90a-41eb-88a0-17178af95977-dshm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:36:30.084164 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:30.084162    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3fef4ca-c90a-41eb-88a0-17178af95977-tls-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:36:30.084164 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:30.084172    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5bbj\" (UniqueName: \"kubernetes.io/projected/e3fef4ca-c90a-41eb-88a0-17178af95977-kube-api-access-w5bbj\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:36:30.891644 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:30.891607    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8" event={"ID":"e3fef4ca-c90a-41eb-88a0-17178af95977","Type":"ContainerDied","Data":"8b074d307eb5225a5e98ea9e1c657586250d1efc7fdfa11d989dd77c47d65fc6"}
Sep 17 23:36:30.892074 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:30.891655    2568 scope.go:117] "RemoveContainer" containerID="05b09ca3c71780a1cb0fbf64a3826173f5c0b56d2e759724150eab31dca17ffb"
Sep 17 23:36:30.892074 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:30.891618    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"
Sep 17 23:36:30.908571 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:30.908546    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"]
Sep 17 23:36:30.912701 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:30.912679    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f80c49/rate-test-model-f80c49-kserve-665fbc6887-mfmn8"]
Sep 17 23:36:31.989664 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:31.989620    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3fef4ca-c90a-41eb-88a0-17178af95977" path="/var/lib/kubelet/pods/e3fef4ca-c90a-41eb-88a0-17178af95977/volumes"
Sep 17 23:36:43.111645 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:43.111599    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:44.339948 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:44.339914    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:44.847918 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:44.847863    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:44.952382 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:44.952351    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:45.102517 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:45.102448    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:45.537777 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:45.537749    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:45.799890 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:45.799793    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:47.700682 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:47.700645    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:48.359510 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:48.359469    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:48.465355 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:48.465326    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:48.566998 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:48.566966    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:49.446967 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:49.446933    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:50.206037 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.206006    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:50.903376 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.903339    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"]
Sep 17 23:36:50.903888 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.903771    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f8f664-61f7-42be-95a3-743158705489" containerName="main"
Sep 17 23:36:50.903888 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.903784    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f8f664-61f7-42be-95a3-743158705489" containerName="main"
Sep 17 23:36:50.903888 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.903800    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3fef4ca-c90a-41eb-88a0-17178af95977" containerName="main"
Sep 17 23:36:50.903888 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.903808    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3fef4ca-c90a-41eb-88a0-17178af95977" containerName="main"
Sep 17 23:36:50.904122 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.903895    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f8f664-61f7-42be-95a3-743158705489" containerName="main"
Sep 17 23:36:50.904122 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.903905    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3fef4ca-c90a-41eb-88a0-17178af95977" containerName="main"
Sep 17 23:36:50.908169 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.908147    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:50.910786 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.910765    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3c1ddb-serving-cert\""
Sep 17 23:36:50.919235 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.919212    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"]
Sep 17 23:36:50.948235 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.948209    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:50.973465 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.973437    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6swjq\" (UniqueName: \"kubernetes.io/projected/f79bf087-fe10-46c2-9c4b-64d802f62661-kube-api-access-6swjq\") pod \"maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:50.973592 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.973497    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79bf087-fe10-46c2-9c4b-64d802f62661-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:50.973643 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.973585    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79bf087-fe10-46c2-9c4b-64d802f62661-metrics-certs\") pod \"maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:50.973643 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:50.973624    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79bf087-fe10-46c2-9c4b-64d802f62661-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:51.055686 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:51.055651    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:51.074948 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:51.074920    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6swjq\" (UniqueName: \"kubernetes.io/projected/f79bf087-fe10-46c2-9c4b-64d802f62661-kube-api-access-6swjq\") pod \"maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:51.075173 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:51.075152    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79bf087-fe10-46c2-9c4b-64d802f62661-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:51.075254 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:51.075218    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79bf087-fe10-46c2-9c4b-64d802f62661-metrics-certs\") pod \"maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:51.075315 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:51.075253    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79bf087-fe10-46c2-9c4b-64d802f62661-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:51.075373 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:36:51.075312    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-serving-cert: secret "maas-api-e2e-ipp-b-3c1ddb-serving-cert" not found
Sep 17 23:36:51.075440 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:36:51.075377    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79bf087-fe10-46c2-9c4b-64d802f62661-maas-api-tls podName:f79bf087-fe10-46c2-9c4b-64d802f62661 nodeName:}" failed. No retries permitted until 2026-09-17 23:36:51.575359839 +0000 UTC m=+1809.799035571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f79bf087-fe10-46c2-9c4b-64d802f62661-maas-api-tls") pod "maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87" (UID: "f79bf087-fe10-46c2-9c4b-64d802f62661") : secret "maas-api-e2e-ipp-b-3c1ddb-serving-cert" not found
Sep 17 23:36:51.075983 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:51.075956    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79bf087-fe10-46c2-9c4b-64d802f62661-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:51.077621 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:51.077601    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79bf087-fe10-46c2-9c4b-64d802f62661-metrics-certs\") pod \"maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:51.088908 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:51.088885    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6swjq\" (UniqueName: \"kubernetes.io/projected/f79bf087-fe10-46c2-9c4b-64d802f62661-kube-api-access-6swjq\") pod \"maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:51.230561 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:51.230479    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:51.443798 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:51.443764    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:51.542268 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:51.542234    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:51.580011 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:51.579981    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79bf087-fe10-46c2-9c4b-64d802f62661-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:51.582578 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:51.582556    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79bf087-fe10-46c2-9c4b-64d802f62661-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:51.821968 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:51.821844    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:52.096567 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:52.096534    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"]
Sep 17 23:36:52.099488 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:36:52.099460    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79bf087_fe10_46c2_9c4b_64d802f62661.slice/crio-68f914279f681a1f13f3fadfaf7a2f44d61903b91cfc07474f4d4d01ba70c710 WatchSource:0}: Error finding container 68f914279f681a1f13f3fadfaf7a2f44d61903b91cfc07474f4d4d01ba70c710: Status 404 returned error can't find the container with id 68f914279f681a1f13f3fadfaf7a2f44d61903b91cfc07474f4d4d01ba70c710
Sep 17 23:36:52.101070 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:52.101048    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:36:53.004861 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:53.004772    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87" event={"ID":"f79bf087-fe10-46c2-9c4b-64d802f62661","Type":"ContainerStarted","Data":"55f967be576cac6b913e710ca8ef3bfee71bbcd63c0e05bc9102e7f45e53e733"}
Sep 17 23:36:53.004861 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:53.004824    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87" event={"ID":"f79bf087-fe10-46c2-9c4b-64d802f62661","Type":"ContainerStarted","Data":"68f914279f681a1f13f3fadfaf7a2f44d61903b91cfc07474f4d4d01ba70c710"}
Sep 17 23:36:53.005100 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:53.004909    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:36:53.030267 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:53.030217    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87" podStartSLOduration=2.501481753 podStartE2EDuration="3.030200164s" totalImagesPullingTime="528.718411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:36:53.028026577 +0000 UTC m=+1811.251702353" watchObservedRunningTime="2026-09-17 23:36:53.030200164 +0000 UTC m=+1811.253875912"
Sep 17 23:36:59.015025 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:36:59.014994    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:37:13.031129 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:13.031097    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:13.354271 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:13.354184    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:13.819550 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:13.819519    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:14.503768 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:14.503733    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:15.353137 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:15.353102    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:16.035912 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:16.035854    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:23.704800 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.704763    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"]
Sep 17 23:37:23.708221 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.708201    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.710934 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.710904    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-36fb5c\"/\"kube-root-ca.crt\""
Sep 17 23:37:23.711855 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.711834    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-36fb5c\"/\"default-dockercfg-9qhdr\""
Sep 17 23:37:23.711942 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.711891    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-36fb5c\"/\"ipp-route-36fb5c-kserve-self-signed-certs\""
Sep 17 23:37:23.711994 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.711891    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-36fb5c\"/\"openshift-service-ca.crt\""
Sep 17 23:37:23.716679 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.716661    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"]
Sep 17 23:37:23.783270 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.783237    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngx7k\" (UniqueName: \"kubernetes.io/projected/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-kube-api-access-ngx7k\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.783459 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.783281    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-home\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.783459 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.783304    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-model-cache\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.783459 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.783367    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-tmp-dir\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.783459 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.783418    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-tls-certs\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.783710 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.783479    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-dshm\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.804154 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.804125    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:23.884507 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.884469    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-tmp-dir\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.884674 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.884546    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-tls-certs\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.884674 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.884579    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-dshm\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.884674 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.884612    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngx7k\" (UniqueName: \"kubernetes.io/projected/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-kube-api-access-ngx7k\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.884674 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.884644    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-home\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.884674 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.884667    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-model-cache\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.885035 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.885006    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-tmp-dir\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.885100 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.885048    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-home\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.885135 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.885099    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-model-cache\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.886964 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.886941    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-dshm\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.887137 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.887119    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-tls-certs\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:23.892201 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:23.892180    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngx7k\" (UniqueName: \"kubernetes.io/projected/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-kube-api-access-ngx7k\") pod \"ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") " pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:24.018400 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:24.018314    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:24.287856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:24.287829    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"]
Sep 17 23:37:24.289844 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:37:24.289818    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e6b649a_ef9f_49f9_a524_99b9c8c3ff1b.slice/crio-5c1c959c08ad432c7e3c9c8a3c28cf737741c729fd7d60459900151e90202350 WatchSource:0}: Error finding container 5c1c959c08ad432c7e3c9c8a3c28cf737741c729fd7d60459900151e90202350: Status 404 returned error can't find the container with id 5c1c959c08ad432c7e3c9c8a3c28cf737741c729fd7d60459900151e90202350
Sep 17 23:37:25.157048 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:25.157014    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr" event={"ID":"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b","Type":"ContainerStarted","Data":"a7a584e3ae91b62b4853f35be2be3efbf831a53e088db47471b1ca6916db23e8"}
Sep 17 23:37:25.157423 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:25.157057    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr" event={"ID":"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b","Type":"ContainerStarted","Data":"5c1c959c08ad432c7e3c9c8a3c28cf737741c729fd7d60459900151e90202350"}
Sep 17 23:37:25.176343 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:25.176290    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr" podStartSLOduration=2.176275026 podStartE2EDuration="2.176275026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:37:25.174012138 +0000 UTC m=+1843.397687887" watchObservedRunningTime="2026-09-17 23:37:25.176275026 +0000 UTC m=+1843.399950775"
Sep 17 23:37:34.019294 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:34.019265    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:34.019294 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:34.019306    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:34.031679 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:34.031652    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:34.211312 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:34.211277    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:37:39.736251 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:39.736216    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:43.322905 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:43.322854    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:55.357734 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:55.357656    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:55.653075 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:55.652990    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:56.111440 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:56.111403    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:58.136678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:58.136647    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:58.804738 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:58.804708    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:58.900401 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:58.900357    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:59.649525 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:37:59.649489    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:00.440704 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:00.440672    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:01.247999 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:01.247965    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:01.899638 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:01.899606    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"]
Sep 17 23:38:01.903727 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:01.903701    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:01.906280 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:01.906255    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-26b61a-serving-cert\""
Sep 17 23:38:01.915195 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:01.915167    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"]
Sep 17 23:38:01.933278 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:01.933243    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:01.945402 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:01.945370    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:01.945616 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:01.945411    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:01.945616 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:01.945479    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfkfl\" (UniqueName: \"kubernetes.io/projected/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-kube-api-access-rfkfl\") pod \"maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:01.945616 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:01.945551    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:02.035093 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:02.035062    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:02.046345 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:02.046315    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:02.046496 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:02.046370    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfkfl\" (UniqueName: \"kubernetes.io/projected/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-kube-api-access-rfkfl\") pod \"maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:02.046496 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:02.046444    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:02.046496 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:38:02.046458    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-serving-cert: secret "maas-api-e2e-ipp-cleanup-26b61a-serving-cert" not found
Sep 17 23:38:02.046627 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:38:02.046522    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-maas-api-tls podName:9d26a1dc-dea2-4280-8a8d-860ffefd7c98 nodeName:}" failed. No retries permitted until 2026-09-17 23:38:02.546500348 +0000 UTC m=+1880.770176088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6" (UID: "9d26a1dc-dea2-4280-8a8d-860ffefd7c98") : secret "maas-api-e2e-ipp-cleanup-26b61a-serving-cert" not found
Sep 17 23:38:02.046627 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:02.046552    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:02.047228 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:02.047206    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:02.049033 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:02.049006    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:02.054982 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:02.054956    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfkfl\" (UniqueName: \"kubernetes.io/projected/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-kube-api-access-rfkfl\") pod \"maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:02.275519 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:02.275478    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:02.387633 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:02.387605    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:02.488502 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:02.488463    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:02.551661 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:02.551562    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:02.554049 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:02.554027    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:02.816581 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:02.816493    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:03.087962 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:03.087935    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"]
Sep 17 23:38:03.089819 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:38:03.089789    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d26a1dc_dea2_4280_8a8d_860ffefd7c98.slice/crio-33ece5fb22db598ea3fa135130e33682337a84cea15222c262d6389f1320299c WatchSource:0}: Error finding container 33ece5fb22db598ea3fa135130e33682337a84cea15222c262d6389f1320299c: Status 404 returned error can't find the container with id 33ece5fb22db598ea3fa135130e33682337a84cea15222c262d6389f1320299c
Sep 17 23:38:03.345748 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:03.345660    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6" event={"ID":"9d26a1dc-dea2-4280-8a8d-860ffefd7c98","Type":"ContainerStarted","Data":"33ece5fb22db598ea3fa135130e33682337a84cea15222c262d6389f1320299c"}
Sep 17 23:38:04.352166 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:04.352127    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6" event={"ID":"9d26a1dc-dea2-4280-8a8d-860ffefd7c98","Type":"ContainerStarted","Data":"523cad65c4ca8fd821111318d8f534e33f180a4fc9689efa83a9a8d78793bcd4"}
Sep 17 23:38:04.352554 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:04.352292    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:04.372201 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:04.372151    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6" podStartSLOduration=2.70913039 podStartE2EDuration="3.372136676s" totalImagesPullingTime="663.006286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:38:04.370737633 +0000 UTC m=+1882.594413382" watchObservedRunningTime="2026-09-17 23:38:04.372136676 +0000 UTC m=+1882.595812423"
Sep 17 23:38:10.362731 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:10.362696    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:24.930424 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:24.930370    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:25.741141 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:25.741104    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:26.434830 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:26.434798    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:27.148847 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:27.148806    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:27.956550 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:27.956518    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:28.643838 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:28.643808    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:28.977829 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:28.977750    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:30.633295 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:30.633264    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:31.419054 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:31.419020    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:32.154902 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:32.154854    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:32.855899 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:32.855854    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:33.552034 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:33.551998    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:34.203127 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:34.203092    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:40.658309 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:40.658274    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"]
Sep 17 23:38:40.658747 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:40.658508    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6" podUID="9d26a1dc-dea2-4280-8a8d-860ffefd7c98" containerName="maas-api" containerID="cri-o://523cad65c4ca8fd821111318d8f534e33f180a4fc9689efa83a9a8d78793bcd4" gracePeriod=30
Sep 17 23:38:40.689230 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:40.689193    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:40.789490 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:40.789461    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:40.888023 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:40.887987    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:40.899039 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:40.899018    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:40.953704 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:40.953602    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:41.019197 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.019161    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-maas-api-tls\") pod \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") "
Sep 17 23:38:41.019388 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.019207    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfkfl\" (UniqueName: \"kubernetes.io/projected/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-kube-api-access-rfkfl\") pod \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") "
Sep 17 23:38:41.019388 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.019322    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-openshift-service-ca\") pod \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") "
Sep 17 23:38:41.019509 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.019405    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-metrics-certs\") pod \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\" (UID: \"9d26a1dc-dea2-4280-8a8d-860ffefd7c98\") "
Sep 17 23:38:41.019786 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.019756    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d26a1dc-dea2-4280-8a8d-860ffefd7c98" (UID: "9d26a1dc-dea2-4280-8a8d-860ffefd7c98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:38:41.021420 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.021388    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9d26a1dc-dea2-4280-8a8d-860ffefd7c98" (UID: "9d26a1dc-dea2-4280-8a8d-860ffefd7c98"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:38:41.021513 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.021438    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-kube-api-access-rfkfl" (OuterVolumeSpecName: "kube-api-access-rfkfl") pod "9d26a1dc-dea2-4280-8a8d-860ffefd7c98" (UID: "9d26a1dc-dea2-4280-8a8d-860ffefd7c98"). InnerVolumeSpecName "kube-api-access-rfkfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:38:41.021513 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.021484    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d26a1dc-dea2-4280-8a8d-860ffefd7c98" (UID: "9d26a1dc-dea2-4280-8a8d-860ffefd7c98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:38:41.120636 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.120602    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:38:41.120636 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.120629    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:38:41.120636 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.120638    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfkfl\" (UniqueName: \"kubernetes.io/projected/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-kube-api-access-rfkfl\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:38:41.120915 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.120648    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d26a1dc-dea2-4280-8a8d-860ffefd7c98-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:38:41.288527 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.288479    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:41.425295 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.425263    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:41.541230 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.541143    2568 generic.go:356] "Generic (PLEG): container finished" podID="9d26a1dc-dea2-4280-8a8d-860ffefd7c98" containerID="523cad65c4ca8fd821111318d8f534e33f180a4fc9689efa83a9a8d78793bcd4" exitCode=0
Sep 17 23:38:41.541230 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.541185    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6" event={"ID":"9d26a1dc-dea2-4280-8a8d-860ffefd7c98","Type":"ContainerDied","Data":"523cad65c4ca8fd821111318d8f534e33f180a4fc9689efa83a9a8d78793bcd4"}
Sep 17 23:38:41.541230 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.541190    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"
Sep 17 23:38:41.541230 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.541227    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6" event={"ID":"9d26a1dc-dea2-4280-8a8d-860ffefd7c98","Type":"ContainerDied","Data":"33ece5fb22db598ea3fa135130e33682337a84cea15222c262d6389f1320299c"}
Sep 17 23:38:41.541514 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.541251    2568 scope.go:117] "RemoveContainer" containerID="523cad65c4ca8fd821111318d8f534e33f180a4fc9689efa83a9a8d78793bcd4"
Sep 17 23:38:41.551204 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.551184    2568 scope.go:117] "RemoveContainer" containerID="523cad65c4ca8fd821111318d8f534e33f180a4fc9689efa83a9a8d78793bcd4"
Sep 17 23:38:41.551457 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:38:41.551440    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523cad65c4ca8fd821111318d8f534e33f180a4fc9689efa83a9a8d78793bcd4\": container with ID starting with 523cad65c4ca8fd821111318d8f534e33f180a4fc9689efa83a9a8d78793bcd4 not found: ID does not exist" containerID="523cad65c4ca8fd821111318d8f534e33f180a4fc9689efa83a9a8d78793bcd4"
Sep 17 23:38:41.551547 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.551464    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523cad65c4ca8fd821111318d8f534e33f180a4fc9689efa83a9a8d78793bcd4"} err="failed to get container status \"523cad65c4ca8fd821111318d8f534e33f180a4fc9689efa83a9a8d78793bcd4\": rpc error: code = NotFound desc = could not find container \"523cad65c4ca8fd821111318d8f534e33f180a4fc9689efa83a9a8d78793bcd4\": container with ID starting with 523cad65c4ca8fd821111318d8f534e33f180a4fc9689efa83a9a8d78793bcd4 not found: ID does not exist"
Sep 17 23:38:41.564254 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.564231    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"]
Sep 17 23:38:41.567782 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.567764    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-26b61a-56479b5684-fwzm6"]
Sep 17 23:38:41.990678 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:41.990587    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d26a1dc-dea2-4280-8a8d-860ffefd7c98" path="/var/lib/kubelet/pods/9d26a1dc-dea2-4280-8a8d-860ffefd7c98/volumes"
Sep 17 23:38:42.095225 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:42.095196    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:42.785600 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:42.785565    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:43.744402 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:43.744365    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:44.399950 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:44.399917    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:47.265632 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:47.265593    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:47.327482 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:47.327443    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:47.557387 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:47.557356    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:51.604607 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:51.604558    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:52.536311 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:52.536254    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:53.237461 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:53.237397    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:53.989140 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:53.989105    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:54.672918 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:54.672881    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:55.323071 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:38:55.323032    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:06.779209 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:06.779174    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:06.851302 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:06.851200    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:06.914036 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:06.914000    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:06.979372 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:06.979341    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:07.352121 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:07.352095    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:07.452130 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:07.452091    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:08.134803 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:08.134772    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:09.042188 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:09.042163    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:09.747357 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:09.747323    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:12.331603 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:12.331566    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"]
Sep 17 23:39:12.373208 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:12.373177    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:27.350918 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.350842    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr" podUID="2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b" containerName="main" containerID="cri-o://a7a584e3ae91b62b4853f35be2be3efbf831a53e088db47471b1ca6916db23e8" gracePeriod=45
Sep 17 23:39:27.602816 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.602753    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:39:27.663758 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.663730    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-model-cache\") pod \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") "
Sep 17 23:39:27.663972 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.663778    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngx7k\" (UniqueName: \"kubernetes.io/projected/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-kube-api-access-ngx7k\") pod \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") "
Sep 17 23:39:27.663972 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.663803    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-home\") pod \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") "
Sep 17 23:39:27.663972 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.663845    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-dshm\") pod \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") "
Sep 17 23:39:27.663972 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.663935    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-tls-certs\") pod \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") "
Sep 17 23:39:27.664198 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.663975    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-tmp-dir\") pod \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\" (UID: \"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b\") "
Sep 17 23:39:27.664198 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.664041    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b" (UID: "2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:27.664308 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.664284    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b" (UID: "2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:27.664361 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.664303    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-home" (OuterVolumeSpecName: "home") pod "2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b" (UID: "2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:27.664361 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.664312    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-model-cache\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:39:27.666079 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.666054    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b" (UID: "2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:39:27.666187 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.666070    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-kube-api-access-ngx7k" (OuterVolumeSpecName: "kube-api-access-ngx7k") pod "2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b" (UID: "2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b"). InnerVolumeSpecName "kube-api-access-ngx7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:39:27.666350 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.666327    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-dshm" (OuterVolumeSpecName: "dshm") pod "2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b" (UID: "2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:27.769786 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.769755    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngx7k\" (UniqueName: \"kubernetes.io/projected/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-kube-api-access-ngx7k\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:39:27.769786 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.769783    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-home\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:39:27.769786 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.769794    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-dshm\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:39:27.770089 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.769801    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-tls-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:39:27.770089 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.769811    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b-tmp-dir\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:39:27.770733 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.770704    2568 generic.go:356] "Generic (PLEG): container finished" podID="2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b" containerID="a7a584e3ae91b62b4853f35be2be3efbf831a53e088db47471b1ca6916db23e8" exitCode=0
Sep 17 23:39:27.770836 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.770747    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr" event={"ID":"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b","Type":"ContainerDied","Data":"a7a584e3ae91b62b4853f35be2be3efbf831a53e088db47471b1ca6916db23e8"}
Sep 17 23:39:27.770836 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.770766    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"
Sep 17 23:39:27.770836 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.770778    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr" event={"ID":"2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b","Type":"ContainerDied","Data":"5c1c959c08ad432c7e3c9c8a3c28cf737741c729fd7d60459900151e90202350"}
Sep 17 23:39:27.770836 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.770795    2568 scope.go:117] "RemoveContainer" containerID="a7a584e3ae91b62b4853f35be2be3efbf831a53e088db47471b1ca6916db23e8"
Sep 17 23:39:27.779912 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.779895    2568 scope.go:117] "RemoveContainer" containerID="a7a584e3ae91b62b4853f35be2be3efbf831a53e088db47471b1ca6916db23e8"
Sep 17 23:39:27.780180 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:39:27.780164    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a584e3ae91b62b4853f35be2be3efbf831a53e088db47471b1ca6916db23e8\": container with ID starting with a7a584e3ae91b62b4853f35be2be3efbf831a53e088db47471b1ca6916db23e8 not found: ID does not exist" containerID="a7a584e3ae91b62b4853f35be2be3efbf831a53e088db47471b1ca6916db23e8"
Sep 17 23:39:27.780220 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.780188    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a584e3ae91b62b4853f35be2be3efbf831a53e088db47471b1ca6916db23e8"} err="failed to get container status \"a7a584e3ae91b62b4853f35be2be3efbf831a53e088db47471b1ca6916db23e8\": rpc error: code = NotFound desc = could not find container \"a7a584e3ae91b62b4853f35be2be3efbf831a53e088db47471b1ca6916db23e8\": container with ID starting with a7a584e3ae91b62b4853f35be2be3efbf831a53e088db47471b1ca6916db23e8 not found: ID does not exist"
Sep 17 23:39:27.791284 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.791260    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"]
Sep 17 23:39:27.793641 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.793620    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-36fb5c/ipp-route-36fb5c-kserve-7d4f9d5759-6fcsr"]
Sep 17 23:39:27.991354 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:27.991281    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b" path="/var/lib/kubelet/pods/2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b/volumes"
Sep 17 23:39:29.637907 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:29.637853    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:48.529816 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:48.529786    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:49.384163 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:49.384130    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:54.793143 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:39:54.793109    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:00.562035 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:00.561999    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:00.628587 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:00.628559    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:00.836654 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:00.836578    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:01.539326 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:01.539296    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:02.443363 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:02.443332    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:03.125699 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:03.125664    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:03.764615 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:03.764587    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:04.547042 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:04.547009    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:11.871757 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:11.871716    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"]
Sep 17 23:40:11.872194 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:11.871932    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87" podUID="f79bf087-fe10-46c2-9c4b-64d802f62661" containerName="maas-api" containerID="cri-o://55f967be576cac6b913e710ca8ef3bfee71bbcd63c0e05bc9102e7f45e53e733" gracePeriod=30
Sep 17 23:40:11.911482 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:11.911451    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:12.004832 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.004787    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:12.105841 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.105810    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:12.121377 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.121338    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:40:12.184944 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.184839    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79bf087-fe10-46c2-9c4b-64d802f62661-metrics-certs\") pod \"f79bf087-fe10-46c2-9c4b-64d802f62661\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") "
Sep 17 23:40:12.184944 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.184931    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79bf087-fe10-46c2-9c4b-64d802f62661-maas-api-tls\") pod \"f79bf087-fe10-46c2-9c4b-64d802f62661\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") "
Sep 17 23:40:12.185174 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.184976    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6swjq\" (UniqueName: \"kubernetes.io/projected/f79bf087-fe10-46c2-9c4b-64d802f62661-kube-api-access-6swjq\") pod \"f79bf087-fe10-46c2-9c4b-64d802f62661\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") "
Sep 17 23:40:12.185174 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.185042    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79bf087-fe10-46c2-9c4b-64d802f62661-openshift-service-ca\") pod \"f79bf087-fe10-46c2-9c4b-64d802f62661\" (UID: \"f79bf087-fe10-46c2-9c4b-64d802f62661\") "
Sep 17 23:40:12.185405 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.185379    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79bf087-fe10-46c2-9c4b-64d802f62661-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f79bf087-fe10-46c2-9c4b-64d802f62661" (UID: "f79bf087-fe10-46c2-9c4b-64d802f62661"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:40:12.186962 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.186930    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79bf087-fe10-46c2-9c4b-64d802f62661-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f79bf087-fe10-46c2-9c4b-64d802f62661" (UID: "f79bf087-fe10-46c2-9c4b-64d802f62661"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:40:12.187080 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.186975    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79bf087-fe10-46c2-9c4b-64d802f62661-kube-api-access-6swjq" (OuterVolumeSpecName: "kube-api-access-6swjq") pod "f79bf087-fe10-46c2-9c4b-64d802f62661" (UID: "f79bf087-fe10-46c2-9c4b-64d802f62661"). InnerVolumeSpecName "kube-api-access-6swjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:40:12.187080 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.186996    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79bf087-fe10-46c2-9c4b-64d802f62661-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f79bf087-fe10-46c2-9c4b-64d802f62661" (UID: "f79bf087-fe10-46c2-9c4b-64d802f62661"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:40:12.204449 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.204426    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:12.286746 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.286722    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79bf087-fe10-46c2-9c4b-64d802f62661-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:40:12.286746 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.286746    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79bf087-fe10-46c2-9c4b-64d802f62661-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:40:12.286927 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.286756    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79bf087-fe10-46c2-9c4b-64d802f62661-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:40:12.286927 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.286765    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6swjq\" (UniqueName: \"kubernetes.io/projected/f79bf087-fe10-46c2-9c4b-64d802f62661-kube-api-access-6swjq\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:40:12.581831 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.581797    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:12.704497 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.704463    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:12.993692 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.993596    2568 generic.go:356] "Generic (PLEG): container finished" podID="f79bf087-fe10-46c2-9c4b-64d802f62661" containerID="55f967be576cac6b913e710ca8ef3bfee71bbcd63c0e05bc9102e7f45e53e733" exitCode=0
Sep 17 23:40:12.993692 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.993651    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"
Sep 17 23:40:12.994212 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.993682    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87" event={"ID":"f79bf087-fe10-46c2-9c4b-64d802f62661","Type":"ContainerDied","Data":"55f967be576cac6b913e710ca8ef3bfee71bbcd63c0e05bc9102e7f45e53e733"}
Sep 17 23:40:12.994212 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.993742    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87" event={"ID":"f79bf087-fe10-46c2-9c4b-64d802f62661","Type":"ContainerDied","Data":"68f914279f681a1f13f3fadfaf7a2f44d61903b91cfc07474f4d4d01ba70c710"}
Sep 17 23:40:12.994212 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:12.993774    2568 scope.go:117] "RemoveContainer" containerID="55f967be576cac6b913e710ca8ef3bfee71bbcd63c0e05bc9102e7f45e53e733"
Sep 17 23:40:13.003797 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:13.003780    2568 scope.go:117] "RemoveContainer" containerID="55f967be576cac6b913e710ca8ef3bfee71bbcd63c0e05bc9102e7f45e53e733"
Sep 17 23:40:13.004126 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:40:13.004106    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f967be576cac6b913e710ca8ef3bfee71bbcd63c0e05bc9102e7f45e53e733\": container with ID starting with 55f967be576cac6b913e710ca8ef3bfee71bbcd63c0e05bc9102e7f45e53e733 not found: ID does not exist" containerID="55f967be576cac6b913e710ca8ef3bfee71bbcd63c0e05bc9102e7f45e53e733"
Sep 17 23:40:13.004204 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:13.004137    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f967be576cac6b913e710ca8ef3bfee71bbcd63c0e05bc9102e7f45e53e733"} err="failed to get container status \"55f967be576cac6b913e710ca8ef3bfee71bbcd63c0e05bc9102e7f45e53e733\": rpc error: code = NotFound desc = could not find container \"55f967be576cac6b913e710ca8ef3bfee71bbcd63c0e05bc9102e7f45e53e733\": container with ID starting with 55f967be576cac6b913e710ca8ef3bfee71bbcd63c0e05bc9102e7f45e53e733 not found: ID does not exist"
Sep 17 23:40:13.016567 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:13.016540    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"]
Sep 17 23:40:13.021962 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:13.021940    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3c1ddb-7b4899c6d7-cbx87"]
Sep 17 23:40:13.573181 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:13.573148    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:13.990294 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:13.990222    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f79bf087-fe10-46c2-9c4b-64d802f62661" path="/var/lib/kubelet/pods/f79bf087-fe10-46c2-9c4b-64d802f62661/volumes"
Sep 17 23:40:14.469081 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:14.469055    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:18.347350 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:18.347319    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:18.447819 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:18.447785    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:18.632580 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:18.632500    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:19.374299 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:19.374267    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:20.245298 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:20.245259    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:20.938586 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:20.938547    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:21.600617 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:21.600563    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:29.550484 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.550442    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"]
Sep 17 23:40:29.551021 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.550710    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls" podUID="e7335ca2-98c3-4524-a488-8306e0042df8" containerName="maas-api" containerID="cri-o://2bebd88a66bf8b07e9b1fe6bb1c283911cdc6c934509e4720584018bb5cbde0d" gracePeriod=30
Sep 17 23:40:29.580279 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.580254    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:29.649088 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.649056    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:29.706951 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.706919    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:29.777264 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.777228    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:29.791878 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.791850    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:40:29.837805 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.837743    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:29.954144 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.954117    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7335ca2-98c3-4524-a488-8306e0042df8-openshift-service-ca\") pod \"e7335ca2-98c3-4524-a488-8306e0042df8\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") "
Sep 17 23:40:29.954308 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.954150    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7335ca2-98c3-4524-a488-8306e0042df8-metrics-certs\") pod \"e7335ca2-98c3-4524-a488-8306e0042df8\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") "
Sep 17 23:40:29.954308 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.954170    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7335ca2-98c3-4524-a488-8306e0042df8-maas-api-tls\") pod \"e7335ca2-98c3-4524-a488-8306e0042df8\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") "
Sep 17 23:40:29.954450 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.954311    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gmcg\" (UniqueName: \"kubernetes.io/projected/e7335ca2-98c3-4524-a488-8306e0042df8-kube-api-access-8gmcg\") pod \"e7335ca2-98c3-4524-a488-8306e0042df8\" (UID: \"e7335ca2-98c3-4524-a488-8306e0042df8\") "
Sep 17 23:40:29.954502 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.954488    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7335ca2-98c3-4524-a488-8306e0042df8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e7335ca2-98c3-4524-a488-8306e0042df8" (UID: "e7335ca2-98c3-4524-a488-8306e0042df8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:40:29.954697 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.954675    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7335ca2-98c3-4524-a488-8306e0042df8-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:40:29.956368 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.956335    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7335ca2-98c3-4524-a488-8306e0042df8-kube-api-access-8gmcg" (OuterVolumeSpecName: "kube-api-access-8gmcg") pod "e7335ca2-98c3-4524-a488-8306e0042df8" (UID: "e7335ca2-98c3-4524-a488-8306e0042df8"). InnerVolumeSpecName "kube-api-access-8gmcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:40:29.956368 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.956355    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7335ca2-98c3-4524-a488-8306e0042df8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e7335ca2-98c3-4524-a488-8306e0042df8" (UID: "e7335ca2-98c3-4524-a488-8306e0042df8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:40:29.956496 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:29.956338    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7335ca2-98c3-4524-a488-8306e0042df8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e7335ca2-98c3-4524-a488-8306e0042df8" (UID: "e7335ca2-98c3-4524-a488-8306e0042df8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:40:30.055469 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:30.055438    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7335ca2-98c3-4524-a488-8306e0042df8-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:40:30.055469 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:30.055462    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7335ca2-98c3-4524-a488-8306e0042df8-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:40:30.055469 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:30.055472    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gmcg\" (UniqueName: \"kubernetes.io/projected/e7335ca2-98c3-4524-a488-8306e0042df8-kube-api-access-8gmcg\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:40:30.075520 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:30.075490    2568 generic.go:356] "Generic (PLEG): container finished" podID="e7335ca2-98c3-4524-a488-8306e0042df8" containerID="2bebd88a66bf8b07e9b1fe6bb1c283911cdc6c934509e4720584018bb5cbde0d" exitCode=0
Sep 17 23:40:30.075659 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:30.075540    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"
Sep 17 23:40:30.075659 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:30.075578    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls" event={"ID":"e7335ca2-98c3-4524-a488-8306e0042df8","Type":"ContainerDied","Data":"2bebd88a66bf8b07e9b1fe6bb1c283911cdc6c934509e4720584018bb5cbde0d"}
Sep 17 23:40:30.075659 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:30.075628    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls" event={"ID":"e7335ca2-98c3-4524-a488-8306e0042df8","Type":"ContainerDied","Data":"1f659cacb5bf62bdf277374168b9fafb4d100de24821abe01da11e703e0d26b4"}
Sep 17 23:40:30.075659 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:30.075653    2568 scope.go:117] "RemoveContainer" containerID="2bebd88a66bf8b07e9b1fe6bb1c283911cdc6c934509e4720584018bb5cbde0d"
Sep 17 23:40:30.084330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:30.084308    2568 scope.go:117] "RemoveContainer" containerID="2bebd88a66bf8b07e9b1fe6bb1c283911cdc6c934509e4720584018bb5cbde0d"
Sep 17 23:40:30.084625 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:40:30.084602    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bebd88a66bf8b07e9b1fe6bb1c283911cdc6c934509e4720584018bb5cbde0d\": container with ID starting with 2bebd88a66bf8b07e9b1fe6bb1c283911cdc6c934509e4720584018bb5cbde0d not found: ID does not exist" containerID="2bebd88a66bf8b07e9b1fe6bb1c283911cdc6c934509e4720584018bb5cbde0d"
Sep 17 23:40:30.084673 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:30.084634    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bebd88a66bf8b07e9b1fe6bb1c283911cdc6c934509e4720584018bb5cbde0d"} err="failed to get container status \"2bebd88a66bf8b07e9b1fe6bb1c283911cdc6c934509e4720584018bb5cbde0d\": rpc error: code = NotFound desc = could not find container \"2bebd88a66bf8b07e9b1fe6bb1c283911cdc6c934509e4720584018bb5cbde0d\": container with ID starting with 2bebd88a66bf8b07e9b1fe6bb1c283911cdc6c934509e4720584018bb5cbde0d not found: ID does not exist"
Sep 17 23:40:30.100078 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:30.100023    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"]
Sep 17 23:40:30.103710 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:30.103686    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f80c49-7c8fff5564-bp6ls"]
Sep 17 23:40:30.221156 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:30.221120    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:30.317163 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:30.317135    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:30.949494 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:30.949457    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:31.989629 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:31.989597    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7335ca2-98c3-4524-a488-8306e0042df8" path="/var/lib/kubelet/pods/e7335ca2-98c3-4524-a488-8306e0042df8/volumes"
Sep 17 23:40:36.134181 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:36.134148    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:36.232449 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:36.232403    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:36.401758 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:36.401673    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:37.646013 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:37.645982    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:38.325277 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:38.325250    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:46.797242 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:46.797199    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:47.550219 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:47.550183    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:47.646538 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:47.646509    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:47.738246 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:47.738205    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:47.797349 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:47.797317    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:48.107664 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:48.107633    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:53.933354 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:53.933319    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:54.003554 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:54.003512    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:54.186924 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:40:54.186776    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:04.734251 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:42:04.734221    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:19.622193 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:42:19.622157    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:21.202570 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:42:21.202538    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:09.483292 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:43:09.483258    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:09.582130 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:43:09.582103    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:18.624110 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:44:18.624025    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:19.342974 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:44:19.342946    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:19.398380 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:44:19.398345    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:24.373533 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:44:24.373486    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:55.902418 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:44:55.902386    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:45:49.219374 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:45:49.219343    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:45:49.696593 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:45:49.696556    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:45:49.953113 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:45:49.953018    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:45:50.010532 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:45:50.010504    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:46:00.016216 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:00.016179    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:46:19.118087 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:19.118054    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:46:20.830942 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:20.830913    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:46:44.476047 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.476010    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"]
Sep 17 23:46:44.476414 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.476364    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7335ca2-98c3-4524-a488-8306e0042df8" containerName="maas-api"
Sep 17 23:46:44.476414 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.476374    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7335ca2-98c3-4524-a488-8306e0042df8" containerName="maas-api"
Sep 17 23:46:44.476414 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.476390    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d26a1dc-dea2-4280-8a8d-860ffefd7c98" containerName="maas-api"
Sep 17 23:46:44.476414 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.476395    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d26a1dc-dea2-4280-8a8d-860ffefd7c98" containerName="maas-api"
Sep 17 23:46:44.476414 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.476409    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f79bf087-fe10-46c2-9c4b-64d802f62661" containerName="maas-api"
Sep 17 23:46:44.476414 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.476416    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f79bf087-fe10-46c2-9c4b-64d802f62661" containerName="maas-api"
Sep 17 23:46:44.476609 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.476429    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b" containerName="main"
Sep 17 23:46:44.476609 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.476434    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b" containerName="main"
Sep 17 23:46:44.476609 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.476487    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d26a1dc-dea2-4280-8a8d-860ffefd7c98" containerName="maas-api"
Sep 17 23:46:44.476609 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.476496    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7335ca2-98c3-4524-a488-8306e0042df8" containerName="maas-api"
Sep 17 23:46:44.476609 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.476501    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6b649a-ef9f-49f9-a524-99b9c8c3ff1b" containerName="main"
Sep 17 23:46:44.476609 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.476508    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f79bf087-fe10-46c2-9c4b-64d802f62661" containerName="maas-api"
Sep 17 23:46:44.479417 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.479401    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:44.490769 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.490733    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"]
Sep 17 23:46:44.604361 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.604326    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94mt5\" (UniqueName: \"kubernetes.io/projected/5e4d6e4d-fb06-433f-b0fb-589da601b670-kube-api-access-94mt5\") pod \"maas-api-86bf64cdb7-qfn5g\" (UID: \"5e4d6e4d-fb06-433f-b0fb-589da601b670\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:44.604545 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.604382    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e4d6e4d-fb06-433f-b0fb-589da601b670-maas-api-tls\") pod \"maas-api-86bf64cdb7-qfn5g\" (UID: \"5e4d6e4d-fb06-433f-b0fb-589da601b670\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:44.604545 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.604399    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4d6e4d-fb06-433f-b0fb-589da601b670-metrics-certs\") pod \"maas-api-86bf64cdb7-qfn5g\" (UID: \"5e4d6e4d-fb06-433f-b0fb-589da601b670\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:44.604545 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.604464    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e4d6e4d-fb06-433f-b0fb-589da601b670-openshift-service-ca\") pod \"maas-api-86bf64cdb7-qfn5g\" (UID: \"5e4d6e4d-fb06-433f-b0fb-589da601b670\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:44.705207 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.705171    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e4d6e4d-fb06-433f-b0fb-589da601b670-openshift-service-ca\") pod \"maas-api-86bf64cdb7-qfn5g\" (UID: \"5e4d6e4d-fb06-433f-b0fb-589da601b670\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:44.705365 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.705249    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94mt5\" (UniqueName: \"kubernetes.io/projected/5e4d6e4d-fb06-433f-b0fb-589da601b670-kube-api-access-94mt5\") pod \"maas-api-86bf64cdb7-qfn5g\" (UID: \"5e4d6e4d-fb06-433f-b0fb-589da601b670\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:44.705365 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.705312    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e4d6e4d-fb06-433f-b0fb-589da601b670-maas-api-tls\") pod \"maas-api-86bf64cdb7-qfn5g\" (UID: \"5e4d6e4d-fb06-433f-b0fb-589da601b670\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:44.705365 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.705336    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4d6e4d-fb06-433f-b0fb-589da601b670-metrics-certs\") pod \"maas-api-86bf64cdb7-qfn5g\" (UID: \"5e4d6e4d-fb06-433f-b0fb-589da601b670\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:44.705999 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.705972    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e4d6e4d-fb06-433f-b0fb-589da601b670-openshift-service-ca\") pod \"maas-api-86bf64cdb7-qfn5g\" (UID: \"5e4d6e4d-fb06-433f-b0fb-589da601b670\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:44.708055 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.708030    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e4d6e4d-fb06-433f-b0fb-589da601b670-metrics-certs\") pod \"maas-api-86bf64cdb7-qfn5g\" (UID: \"5e4d6e4d-fb06-433f-b0fb-589da601b670\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:44.708292 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.708274    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e4d6e4d-fb06-433f-b0fb-589da601b670-maas-api-tls\") pod \"maas-api-86bf64cdb7-qfn5g\" (UID: \"5e4d6e4d-fb06-433f-b0fb-589da601b670\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:44.713166 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.713146    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94mt5\" (UniqueName: \"kubernetes.io/projected/5e4d6e4d-fb06-433f-b0fb-589da601b670-kube-api-access-94mt5\") pod \"maas-api-86bf64cdb7-qfn5g\" (UID: \"5e4d6e4d-fb06-433f-b0fb-589da601b670\") " pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:44.789813 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:44.789787    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:45.066175 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:45.066137    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"]
Sep 17 23:46:45.067426 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:46:45.067398    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4d6e4d_fb06_433f_b0fb_589da601b670.slice/crio-75909af5c6ba0af0213e42222727bf53be997a36792c1fb92ee07d3e2da017a3 WatchSource:0}: Error finding container 75909af5c6ba0af0213e42222727bf53be997a36792c1fb92ee07d3e2da017a3: Status 404 returned error can't find the container with id 75909af5c6ba0af0213e42222727bf53be997a36792c1fb92ee07d3e2da017a3
Sep 17 23:46:45.068661 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:45.068643    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:46:45.836132 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:45.836102    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g" event={"ID":"5e4d6e4d-fb06-433f-b0fb-589da601b670","Type":"ContainerStarted","Data":"0ea01fa8a008230cb2d6ac0cc4538de63e548d9c8957f7ab0d4b6761a5d7fb4f"}
Sep 17 23:46:45.836521 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:45.836156    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g" event={"ID":"5e4d6e4d-fb06-433f-b0fb-589da601b670","Type":"ContainerStarted","Data":"75909af5c6ba0af0213e42222727bf53be997a36792c1fb92ee07d3e2da017a3"}
Sep 17 23:46:45.836521 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:45.836206    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:45.853679 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:45.853478    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g" podStartSLOduration=1.505600518 podStartE2EDuration="1.853459379s" totalImagesPullingTime="347.858861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:46:45.853056966 +0000 UTC m=+2404.076732717" watchObservedRunningTime="2026-09-17 23:46:45.853459379 +0000 UTC m=+2404.077135129"
Sep 17 23:46:51.852679 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:51.852649    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86bf64cdb7-qfn5g"
Sep 17 23:46:51.896882 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:51.896818    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"]
Sep 17 23:46:51.897092 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:51.897049    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6" podUID="60afdc58-a3b9-4b21-ab79-ffd0ca31801c" containerName="maas-api" containerID="cri-o://9d64d97112ec522e832620290ce03d339d6e98c899d3c30d481751cfc4676ed6" gracePeriod=30
Sep 17 23:46:53.238384 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.238362    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:46:53.279330 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.279304    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-metrics-certs\") pod \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") "
Sep 17 23:46:53.279487 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.279355    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-maas-api-tls\") pod \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") "
Sep 17 23:46:53.279487 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.279417    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-openshift-service-ca\") pod \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") "
Sep 17 23:46:53.279487 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.279435    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78772\" (UniqueName: \"kubernetes.io/projected/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-kube-api-access-78772\") pod \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\" (UID: \"60afdc58-a3b9-4b21-ab79-ffd0ca31801c\") "
Sep 17 23:46:53.279842 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.279816    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60afdc58-a3b9-4b21-ab79-ffd0ca31801c" (UID: "60afdc58-a3b9-4b21-ab79-ffd0ca31801c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:46:53.281558 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.281527    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60afdc58-a3b9-4b21-ab79-ffd0ca31801c" (UID: "60afdc58-a3b9-4b21-ab79-ffd0ca31801c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:46:53.281645 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.281580    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "60afdc58-a3b9-4b21-ab79-ffd0ca31801c" (UID: "60afdc58-a3b9-4b21-ab79-ffd0ca31801c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:46:53.281645 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.281593    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-kube-api-access-78772" (OuterVolumeSpecName: "kube-api-access-78772") pod "60afdc58-a3b9-4b21-ab79-ffd0ca31801c" (UID: "60afdc58-a3b9-4b21-ab79-ffd0ca31801c"). InnerVolumeSpecName "kube-api-access-78772". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:46:53.380115 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.380076    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-maas-api-tls\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:46:53.380115 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.380107    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-openshift-service-ca\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:46:53.380115 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.380117    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78772\" (UniqueName: \"kubernetes.io/projected/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-kube-api-access-78772\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:46:53.380339 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.380126    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60afdc58-a3b9-4b21-ab79-ffd0ca31801c-metrics-certs\") on node \"ip-10-0-140-44.ec2.internal\" DevicePath \"\""
Sep 17 23:46:53.875147 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.875114    2568 generic.go:356] "Generic (PLEG): container finished" podID="60afdc58-a3b9-4b21-ab79-ffd0ca31801c" containerID="9d64d97112ec522e832620290ce03d339d6e98c899d3c30d481751cfc4676ed6" exitCode=0
Sep 17 23:46:53.875348 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.875163    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"
Sep 17 23:46:53.875348 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.875163    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6" event={"ID":"60afdc58-a3b9-4b21-ab79-ffd0ca31801c","Type":"ContainerDied","Data":"9d64d97112ec522e832620290ce03d339d6e98c899d3c30d481751cfc4676ed6"}
Sep 17 23:46:53.875348 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.875296    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6" event={"ID":"60afdc58-a3b9-4b21-ab79-ffd0ca31801c","Type":"ContainerDied","Data":"c39268ecbdfa8a3f2b4efa255b67d6400393fc7190ee31d563b1aa9f063024cd"}
Sep 17 23:46:53.875348 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.875331    2568 scope.go:117] "RemoveContainer" containerID="9d64d97112ec522e832620290ce03d339d6e98c899d3c30d481751cfc4676ed6"
Sep 17 23:46:53.885211 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.885191    2568 scope.go:117] "RemoveContainer" containerID="9d64d97112ec522e832620290ce03d339d6e98c899d3c30d481751cfc4676ed6"
Sep 17 23:46:53.885494 ip-10-0-140-44 kubenswrapper[2568]: E0917 23:46:53.885475    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d64d97112ec522e832620290ce03d339d6e98c899d3c30d481751cfc4676ed6\": container with ID starting with 9d64d97112ec522e832620290ce03d339d6e98c899d3c30d481751cfc4676ed6 not found: ID does not exist" containerID="9d64d97112ec522e832620290ce03d339d6e98c899d3c30d481751cfc4676ed6"
Sep 17 23:46:53.885550 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.885506    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d64d97112ec522e832620290ce03d339d6e98c899d3c30d481751cfc4676ed6"} err="failed to get container status \"9d64d97112ec522e832620290ce03d339d6e98c899d3c30d481751cfc4676ed6\": rpc error: code = NotFound desc = could not find container \"9d64d97112ec522e832620290ce03d339d6e98c899d3c30d481751cfc4676ed6\": container with ID starting with 9d64d97112ec522e832620290ce03d339d6e98c899d3c30d481751cfc4676ed6 not found: ID does not exist"
Sep 17 23:46:53.896209 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.896187    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"]
Sep 17 23:46:53.900332 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.900313    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d54fc9767-4jnj6"]
Sep 17 23:46:53.989920 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:46:53.989893    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60afdc58-a3b9-4b21-ab79-ffd0ca31801c" path="/var/lib/kubelet/pods/60afdc58-a3b9-4b21-ab79-ffd0ca31801c/volumes"
Sep 17 23:47:12.165419 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:12.165366    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697bb5576d-484bm_10f49254-7996-4cbf-859a-cb932d1ef50b/manager/0.log"
Sep 17 23:47:12.171147 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:12.171124    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697bb5576d-484bm_10f49254-7996-4cbf-859a-cb932d1ef50b/copy-manifests/0.log"
Sep 17 23:47:12.859707 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:12.859670    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-86bf64cdb7-qfn5g_5e4d6e4d-fb06-433f-b0fb-589da601b670/maas-api/0.log"
Sep 17 23:47:13.179313 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:13.179241    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-djm4s_12b22b89-bf60-4d6b-bbeb-59d89590dde0/postgres/0.log"
Sep 17 23:47:14.264724 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:14.264693    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9rnfc_405b6ac1-52ac-40c2-90d3-1117943ebb71/kuadrant-console-plugin/0.log"
Sep 17 23:47:15.286060 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:15.286034    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65f4688f79-k7p9x_61fdd960-430c-40fa-8b74-e3bcd05b12c3/kube-auth-proxy/0.log"
Sep 17 23:47:16.183295 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:16.183269    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-5rqzg_0a4e0339-0272-4cfd-92f2-e1aaad8d867b/main/0.log"
Sep 17 23:47:16.399042 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:16.399014    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-q5g6s_bb3255ee-5e56-46ef-a02d-4ba9dce85c2b/main/0.log"
Sep 17 23:47:16.510495 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:16.510424    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fs6lgl_a208e26b-f4d8-4a8c-9dc8-961b0b2c6781/main/0.log"
Sep 17 23:47:16.619451 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:16.619422    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-l6tbq_e1261b14-190b-4e6c-829a-1eab3a3142c0/main/0.log"
Sep 17 23:47:16.728461 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:16.728424    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-tw79f_1f9a8fbf-4cc9-4ed0-bd9f-730d8cb9f0e0/main/0.log"
Sep 17 23:47:22.439439 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:22.439406    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:47:23.593106 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:23.593072    2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log" limit=1024
Sep 17 23:47:25.368653 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:25.368621    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rw5hp_75be2122-acf1-43e5-902d-c1d574d42792/global-pull-secret-syncer/0.log"
Sep 17 23:47:25.412647 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:25.412612    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5xj9x_67f097ec-5a12-42f3-84bf-a66968a7a323/konnectivity-agent/0.log"
Sep 17 23:47:25.531495 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:25.531465    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-44.ec2.internal_551ba4d2dbc3a651cf174938293b65bf/haproxy/0.log"
Sep 17 23:47:30.418400 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:30.418313    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9rnfc_405b6ac1-52ac-40c2-90d3-1117943ebb71/kuadrant-console-plugin/0.log"
Sep 17 23:47:32.294031 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.293970    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-tg8gr_1a866547-9807-43ca-86bd-a50a9d7d0f30/kube-state-metrics/0.log"
Sep 17 23:47:32.324429 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.324352    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-tg8gr_1a866547-9807-43ca-86bd-a50a9d7d0f30/kube-rbac-proxy-main/0.log"
Sep 17 23:47:32.345340 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.345311    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-tg8gr_1a866547-9807-43ca-86bd-a50a9d7d0f30/kube-rbac-proxy-self/0.log"
Sep 17 23:47:32.430218 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.430189    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-55dvl_cd5f437d-1f12-49f2-bfeb-0dda727c740b/node-exporter/0.log"
Sep 17 23:47:32.452856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.452828    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-55dvl_cd5f437d-1f12-49f2-bfeb-0dda727c740b/kube-rbac-proxy/0.log"
Sep 17 23:47:32.475339 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.475312    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-55dvl_cd5f437d-1f12-49f2-bfeb-0dda727c740b/init-textfile/0.log"
Sep 17 23:47:32.693584 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.693556    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-jwstr_80ef4d7a-3b32-4de6-b515-61762093f346/kube-rbac-proxy-main/0.log"
Sep 17 23:47:32.713393 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.713368    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-jwstr_80ef4d7a-3b32-4de6-b515-61762093f346/kube-rbac-proxy-self/0.log"
Sep 17 23:47:32.733662 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.733631    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-jwstr_80ef4d7a-3b32-4de6-b515-61762093f346/openshift-state-metrics/0.log"
Sep 17 23:47:32.766258 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.766227    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39b3f0ef-e598-4742-af00-180e9f1cd9d2/prometheus/0.log"
Sep 17 23:47:32.786114 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.786086    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39b3f0ef-e598-4742-af00-180e9f1cd9d2/config-reloader/0.log"
Sep 17 23:47:32.806659 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.806619    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39b3f0ef-e598-4742-af00-180e9f1cd9d2/thanos-sidecar/0.log"
Sep 17 23:47:32.829626 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.829602    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39b3f0ef-e598-4742-af00-180e9f1cd9d2/kube-rbac-proxy-web/0.log"
Sep 17 23:47:32.853407 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.853334    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39b3f0ef-e598-4742-af00-180e9f1cd9d2/kube-rbac-proxy/0.log"
Sep 17 23:47:32.874897 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.874848    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39b3f0ef-e598-4742-af00-180e9f1cd9d2/kube-rbac-proxy-thanos/0.log"
Sep 17 23:47:32.903275 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.903249    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39b3f0ef-e598-4742-af00-180e9f1cd9d2/init-config-reloader/0.log"
Sep 17 23:47:32.932247 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.932217    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-n2rtl_3415eddf-d7d2-4171-8269-5ca12702a674/prometheus-operator/0.log"
Sep 17 23:47:32.952729 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.952675    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-n2rtl_3415eddf-d7d2-4171-8269-5ca12702a674/kube-rbac-proxy/0.log"
Sep 17 23:47:32.975734 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:32.975705    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-nlgxb_e5dd9ad2-27a9-4a50-9796-8d82601b1228/prometheus-operator-admission-webhook/0.log"
Sep 17 23:47:33.087243 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:33.087190    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-787444c6d5-86h6h_fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9/thanos-query/0.log"
Sep 17 23:47:33.107687 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:33.107585    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-787444c6d5-86h6h_fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9/kube-rbac-proxy-web/0.log"
Sep 17 23:47:33.129443 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:33.129420    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-787444c6d5-86h6h_fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9/kube-rbac-proxy/0.log"
Sep 17 23:47:33.150846 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:33.150821    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-787444c6d5-86h6h_fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9/prom-label-proxy/0.log"
Sep 17 23:47:33.170753 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:33.170730    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-787444c6d5-86h6h_fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9/kube-rbac-proxy-rules/0.log"
Sep 17 23:47:33.190578 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:33.190555    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-787444c6d5-86h6h_fe2140a9-cfbc-43c2-b6ed-ab77fc6184d9/kube-rbac-proxy-metrics/0.log"
Sep 17 23:47:35.932540 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:35.932506    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"]
Sep 17 23:47:35.933058 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:35.932890    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60afdc58-a3b9-4b21-ab79-ffd0ca31801c" containerName="maas-api"
Sep 17 23:47:35.933058 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:35.932906    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="60afdc58-a3b9-4b21-ab79-ffd0ca31801c" containerName="maas-api"
Sep 17 23:47:35.933058 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:35.932969    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="60afdc58-a3b9-4b21-ab79-ffd0ca31801c" containerName="maas-api"
Sep 17 23:47:35.936284 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:35.936267    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:35.938685 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:35.938664    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7zwb4\"/\"openshift-service-ca.crt\""
Sep 17 23:47:35.938685 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:35.938676    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7zwb4\"/\"default-dockercfg-xtqbw\""
Sep 17 23:47:35.939527 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:35.939511    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7zwb4\"/\"kube-root-ca.crt\""
Sep 17 23:47:35.946416 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:35.946394    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"]
Sep 17 23:47:35.967485 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:35.967454    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d6tj\" (UniqueName: \"kubernetes.io/projected/9944c512-22cf-4855-ab24-2f1d58ea8e32-kube-api-access-6d6tj\") pod \"perf-node-gather-daemonset-szc4n\" (UID: \"9944c512-22cf-4855-ab24-2f1d58ea8e32\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:35.967688 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:35.967492    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9944c512-22cf-4855-ab24-2f1d58ea8e32-podres\") pod \"perf-node-gather-daemonset-szc4n\" (UID: \"9944c512-22cf-4855-ab24-2f1d58ea8e32\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:35.967688 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:35.967585    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9944c512-22cf-4855-ab24-2f1d58ea8e32-lib-modules\") pod \"perf-node-gather-daemonset-szc4n\" (UID: \"9944c512-22cf-4855-ab24-2f1d58ea8e32\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:35.967688 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:35.967624    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9944c512-22cf-4855-ab24-2f1d58ea8e32-proc\") pod \"perf-node-gather-daemonset-szc4n\" (UID: \"9944c512-22cf-4855-ab24-2f1d58ea8e32\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:35.967839 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:35.967717    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9944c512-22cf-4855-ab24-2f1d58ea8e32-sys\") pod \"perf-node-gather-daemonset-szc4n\" (UID: \"9944c512-22cf-4855-ab24-2f1d58ea8e32\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:35.989093 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:35.989067    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5f8d85b594-78bzs_81c6cd29-1502-446f-8ac3-4f20bebed00b/console/0.log"
Sep 17 23:47:36.068191 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:36.068150    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9944c512-22cf-4855-ab24-2f1d58ea8e32-sys\") pod \"perf-node-gather-daemonset-szc4n\" (UID: \"9944c512-22cf-4855-ab24-2f1d58ea8e32\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:36.068384 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:36.068200    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d6tj\" (UniqueName: \"kubernetes.io/projected/9944c512-22cf-4855-ab24-2f1d58ea8e32-kube-api-access-6d6tj\") pod \"perf-node-gather-daemonset-szc4n\" (UID: \"9944c512-22cf-4855-ab24-2f1d58ea8e32\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:36.068384 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:36.068238    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9944c512-22cf-4855-ab24-2f1d58ea8e32-podres\") pod \"perf-node-gather-daemonset-szc4n\" (UID: \"9944c512-22cf-4855-ab24-2f1d58ea8e32\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:36.068384 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:36.068277    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9944c512-22cf-4855-ab24-2f1d58ea8e32-sys\") pod \"perf-node-gather-daemonset-szc4n\" (UID: \"9944c512-22cf-4855-ab24-2f1d58ea8e32\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:36.068384 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:36.068303    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9944c512-22cf-4855-ab24-2f1d58ea8e32-lib-modules\") pod \"perf-node-gather-daemonset-szc4n\" (UID: \"9944c512-22cf-4855-ab24-2f1d58ea8e32\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:36.068557 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:36.068424    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9944c512-22cf-4855-ab24-2f1d58ea8e32-lib-modules\") pod \"perf-node-gather-daemonset-szc4n\" (UID: \"9944c512-22cf-4855-ab24-2f1d58ea8e32\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:36.068557 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:36.068423    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9944c512-22cf-4855-ab24-2f1d58ea8e32-podres\") pod \"perf-node-gather-daemonset-szc4n\" (UID: \"9944c512-22cf-4855-ab24-2f1d58ea8e32\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:36.068557 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:36.068465    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9944c512-22cf-4855-ab24-2f1d58ea8e32-proc\") pod \"perf-node-gather-daemonset-szc4n\" (UID: \"9944c512-22cf-4855-ab24-2f1d58ea8e32\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:36.068557 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:36.068542    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9944c512-22cf-4855-ab24-2f1d58ea8e32-proc\") pod \"perf-node-gather-daemonset-szc4n\" (UID: \"9944c512-22cf-4855-ab24-2f1d58ea8e32\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:36.076336 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:36.076310    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d6tj\" (UniqueName: \"kubernetes.io/projected/9944c512-22cf-4855-ab24-2f1d58ea8e32-kube-api-access-6d6tj\") pod \"perf-node-gather-daemonset-szc4n\" (UID: \"9944c512-22cf-4855-ab24-2f1d58ea8e32\") " pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:36.247692 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:36.247609    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:36.517856 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:36.517824    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"]
Sep 17 23:47:36.520778 ip-10-0-140-44 kubenswrapper[2568]: W0917 23:47:36.520716    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9944c512_22cf_4855_ab24_2f1d58ea8e32.slice/crio-f4debc0d3a678646406c47c2e2b3734082c1ef383628db60ab1f9b08e0fc552c WatchSource:0}: Error finding container f4debc0d3a678646406c47c2e2b3734082c1ef383628db60ab1f9b08e0fc552c: Status 404 returned error can't find the container with id f4debc0d3a678646406c47c2e2b3734082c1ef383628db60ab1f9b08e0fc552c
Sep 17 23:47:37.080134 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:37.080100    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n" event={"ID":"9944c512-22cf-4855-ab24-2f1d58ea8e32","Type":"ContainerStarted","Data":"f25a7feb982fa24d1971fc51c924dd498fc2a7cf4c3b0950132fa1bf6b62be93"}
Sep 17 23:47:37.080529 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:37.080156    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n" event={"ID":"9944c512-22cf-4855-ab24-2f1d58ea8e32","Type":"ContainerStarted","Data":"f4debc0d3a678646406c47c2e2b3734082c1ef383628db60ab1f9b08e0fc552c"}
Sep 17 23:47:37.080529 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:37.080242    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:37.096773 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:37.096725    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n" podStartSLOduration=2.096708545 podStartE2EDuration="2.096708545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:47:37.095206547 +0000 UTC m=+2455.318882294" watchObservedRunningTime="2026-09-17 23:47:37.096708545 +0000 UTC m=+2455.320384292"
Sep 17 23:47:37.903087 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:37.903039    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4sv94_1be91cd5-c0cc-4166-9d53-fb219bf61469/dns/0.log"
Sep 17 23:47:37.921854 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:37.921832    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4sv94_1be91cd5-c0cc-4166-9d53-fb219bf61469/kube-rbac-proxy/0.log"
Sep 17 23:47:38.005747 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:38.005713    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qkccv_2d385719-4e1c-4eb3-8866-9fc9b3cdd6aa/dns-node-resolver/0.log"
Sep 17 23:47:38.585488 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:38.585459    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dghj8_f537520e-e943-4837-a871-3142a15f6fba/node-ca/0.log"
Sep 17 23:47:40.115063 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:40.115034    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65f4688f79-k7p9x_61fdd960-430c-40fa-8b74-e3bcd05b12c3/kube-auth-proxy/0.log"
Sep 17 23:47:41.170688 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:41.170648    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wkjrp_677bf714-7a8a-4ce3-8610-5d304d16131e/serve-healthcheck-canary/0.log"
Sep 17 23:47:41.800782 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:41.800746    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wcn4x_89ac468a-64c9-4576-a4a6-998e922cae73/kube-rbac-proxy/0.log"
Sep 17 23:47:41.821692 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:41.821665    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wcn4x_89ac468a-64c9-4576-a4a6-998e922cae73/exporter/0.log"
Sep 17 23:47:41.841425 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:41.841399    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wcn4x_89ac468a-64c9-4576-a4a6-998e922cae73/extractor/0.log"
Sep 17 23:47:43.094399 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:43.094374    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7zwb4/perf-node-gather-daemonset-szc4n"
Sep 17 23:47:44.583862 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:44.583786    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697bb5576d-484bm_10f49254-7996-4cbf-859a-cb932d1ef50b/manager/0.log"
Sep 17 23:47:44.605640 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:44.605611    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697bb5576d-484bm_10f49254-7996-4cbf-859a-cb932d1ef50b/copy-manifests/0.log"
Sep 17 23:47:46.265621 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:46.265546    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-858688bfcc-xs7c6_5ebdf634-ff58-4fba-8b41-03f09f36daf5/manager/0.log"
Sep 17 23:47:46.287876 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:46.287849    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-8w9cg_07dcf854-06eb-4a91-8f2b-8cabdf11478e/openshift-lws-operator/0.log"
Sep 17 23:47:47.231934 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:47.231901    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65f4688f79-k7p9x_61fdd960-430c-40fa-8b74-e3bcd05b12c3/kube-auth-proxy/0.log"
Sep 17 23:47:49.506291 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:49.506261    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65f4688f79-k7p9x_61fdd960-430c-40fa-8b74-e3bcd05b12c3/kube-auth-proxy/0.log"
Sep 17 23:47:52.913383 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:52.913352    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sjsw_13483042-c3be-4ef8-b615-72852899eade/kube-multus-additional-cni-plugins/0.log"
Sep 17 23:47:52.938346 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:52.938312    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sjsw_13483042-c3be-4ef8-b615-72852899eade/egress-router-binary-copy/0.log"
Sep 17 23:47:52.961066 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:52.961030    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sjsw_13483042-c3be-4ef8-b615-72852899eade/cni-plugins/0.log"
Sep 17 23:47:52.982118 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:52.982091    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sjsw_13483042-c3be-4ef8-b615-72852899eade/bond-cni-plugin/0.log"
Sep 17 23:47:53.001972 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:53.001943    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sjsw_13483042-c3be-4ef8-b615-72852899eade/routeoverride-cni/0.log"
Sep 17 23:47:53.022110 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:53.022080    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sjsw_13483042-c3be-4ef8-b615-72852899eade/whereabouts-cni-bincopy/0.log"
Sep 17 23:47:53.043216 ip-10-0-140-44 kubenswrapper[2568]: I0917 23:47:53.043194    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9sjsw_13483042-c3be-4ef8-b615-72852899eade/whereabouts-cni/0.log"