Sep 15 12:23:27.918099 ip-10-0-129-17 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 15 12:23:27.918294 ip-10-0-129-17 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 15 12:23:27.918380 ip-10-0-129-17 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 15 12:23:27.918628 ip-10-0-129-17 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 15 12:23:38.024023 ip-10-0-129-17 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 15 12:23:38.024037 ip-10-0-129-17 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 596bd0444cde4ca59e0d591ecb78c649 --
Sep 15 12:25:28.487161 ip-10-0-129-17 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 12:25:28.932601 ip-10-0-129-17 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 12:25:28.932601 ip-10-0-129-17 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 12:25:28.932601 ip-10-0-129-17 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 12:25:28.932601 ip-10-0-129-17 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 12:25:28.932601 ip-10-0-129-17 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 12:25:28.933696 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.933607    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 12:25:28.940230 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940215    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:25:28.940230 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940230    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940234    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940238    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940241    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940244    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940247    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940250    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940253    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940257    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940260    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940270    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940273    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940276    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940279    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940281    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940284    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940286    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940289    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940292    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940296    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:25:28.940294 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940300    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940303    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940306    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940309    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940312    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940315    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940318    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940320    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940323    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940325    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940329    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940333    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940336    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940338    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940341    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940344    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940346    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940349    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940351    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940354    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:25:28.940766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940356    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940359    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940362    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940365    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940367    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940370    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940372    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940375    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940377    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940380    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940382    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940385    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940387    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940391    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940394    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940397    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940400    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940402    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940404    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940407    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:25:28.941274 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940436    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940440    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940444    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940447    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940450    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940453    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940456    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940459    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940462    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940465    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940468    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940470    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940473    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940476    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940479    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940482    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940485    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940488    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940492    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940494    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:25:28.941766 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940497    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940500    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940503    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940506    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940509    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940511    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940515    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940518    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940521    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940524    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940528    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940532    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940535    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940537    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940540    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940544    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940546    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:25:28.942543 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.940549    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:25:28.943635 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943620    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:25:28.943635 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943635    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943639    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943643    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943646    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943649    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943651    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943654    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943657    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943660    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943663    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943666    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943668    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943671    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943674    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943676    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943679    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943683    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943688    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943691    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:25:28.943695 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943694    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943697    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943700    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943703    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943706    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943712    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943715    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943717    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943720    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943723    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943725    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943728    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943730    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943733    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943736    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943739    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943741    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943744    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943746    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943749    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:25:28.944194 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943751    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943754    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943756    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943758    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943761    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943764    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943766    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943769    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943772    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943774    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943777    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943780    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943782    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943786    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943789    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943792    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943794    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943797    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943800    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943802    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943805    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:25:28.944684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943807    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943810    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943812    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943815    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943817    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943820    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943822    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943825    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943827    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943844    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943847    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943851    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943854    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943857    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943860    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943863    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943865    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943869    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943871    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:25:28.945234 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943875    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943879    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943882    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943886    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943889    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943892    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943896    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943899    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943901    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943904    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943907    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943909    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943912    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943914    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943917    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943919    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943922    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943925    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.943927    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944313    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:25:28.945780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944318    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944320    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944323    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944326    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944329    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944331    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944336    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944340    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944344    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944347    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944351    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944354    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944356    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944359    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944362    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944365    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944368    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944371    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944374    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:25:28.946306 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944377    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944379    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944382    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944385    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944388    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944390    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944393    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944395    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944398    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944400    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944403    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944406    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944408    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944411    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944413    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944418    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944422    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944425    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944427    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:25:28.946803 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944431    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944434    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944437    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944439    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944443    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944446    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944449    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944451    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944454    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944457    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944460    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944462    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944465    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944467    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944470    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944472    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944475    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944477    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944480    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944483    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:25:28.947296 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944485    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944488    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944490    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944493    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944496    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944498    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944501    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944503    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944506    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944509    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944512    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944514    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944517    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944520    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944523    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944526    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944528    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944531    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944534    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944536    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:25:28.947780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944539    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944541    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944544    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944547    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944549    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944552    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944554    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944557    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944559    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944562    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944564    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944567    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944570    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944572    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944575    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944577    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944580    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944582    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944585    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.944587    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:25:28.948324 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944663    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944698    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944704    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944708    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944713    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944716    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944720    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944724    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944727    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944730    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944734    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944737    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944740    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944742    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944745    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944748    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944752    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944754    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944759    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944762    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944765    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944768    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944771    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944774    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 12:25:28.948794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944777    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944780    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944783    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944786    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944789    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944792    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944795    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944798    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944802    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944804    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944807    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944810    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944813    2579 flags.go:64] FLAG: --enable-server="true"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944816    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944820    2579 flags.go:64] FLAG: --event-burst="100"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944823    2579 flags.go:64] FLAG: --event-qps="50"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944825    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944843    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944848    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944854    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944859    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944863    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944866    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944869    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944872    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 12:25:28.949419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944875    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944878    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944884    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944887    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944890    2579 flags.go:64] FLAG: --feature-gates=""
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944894    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944897    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944900    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944903    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944906    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944910    2579 flags.go:64] FLAG: --help="false"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944912    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-129-17.ec2.internal"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944916    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944918    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944921    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944925    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944928    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944931    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944934    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944936    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944939    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944942    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944945    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944948    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 15 12:25:28.950050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944950    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944953    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944956    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944959    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944962    2579 flags.go:64] FLAG: --lock-file=""
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944964    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944967    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944970    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944975    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944980    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944984    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944987    2579 flags.go:64] FLAG: --logging-format="text"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944990    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944993    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944996    2579 flags.go:64] FLAG: --manifest-url=""
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.944998    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945002    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945005    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945009    2579 flags.go:64] FLAG: --max-pods="110"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945012    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945015    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945018    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945021    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945024    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945027    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 12:25:28.950639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945030    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945036    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945039    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945042    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945045    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945048    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945053    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945056    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945059    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945062    2579 flags.go:64] FLAG: --port="10250"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945065    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945068    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d1f3a9d62942334a"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945071    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945074    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945076    2579 flags.go:64] FLAG: --register-node="true"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945079    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945083    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945087    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945090    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945093    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945097    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945100    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945103    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945105    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945108    2579 flags.go:64] FLAG: --runonce="false"
Sep 15 12:25:28.951274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945111    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945114    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945117    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945120    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945123    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945126    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945130    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945134    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945136    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945140    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945142    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945145    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945148    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945151    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945154    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945159    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945162    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945164    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945169    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945171    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945174    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945177    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945180    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945183    2579 flags.go:64] FLAG: --v="2"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945188    2579 flags.go:64] FLAG: --version="false"
Sep 15 12:25:28.951944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945193    2579 flags.go:64] FLAG: --vmodule=""
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945196    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.945200    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945312    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945315    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945319    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945321    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945324    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945327    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945329    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945332    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945335    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945338    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945342    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945344    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945347    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945350    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945352    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945355    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945358    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:25:28.952557 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945360    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945363    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945366    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945368    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945370    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945373    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945376    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945378    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945381    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945384    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945388    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945394    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945397    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945400    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945403    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945405    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945408    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945412    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945415    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:25:28.953069 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945418    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945421    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945424    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945427    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945430    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945432    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945438    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945441    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945444    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945447    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945450    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945453    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945455    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945458    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945460    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945463    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945466    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945468    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945471    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945473    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:25:28.953531 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945476    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945478    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945481    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945484    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945487    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945490    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945493    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945505    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945507    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945510    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945513    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945516    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945518    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945521    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945523    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945526    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945528    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945531    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945535    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945537    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:25:28.954038 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945540    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945543    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945546    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945549    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945551    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945554    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945568    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945572    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945576    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945579    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945582    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945584    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945587    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945590    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945592    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945595    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945599    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945602    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945604    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:25:28.954534 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945607    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:25:28.955015 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945610    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:25:28.955015 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945613    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:25:28.955015 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.945616    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:25:28.955015 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.946281    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 12:25:28.955015 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.954047    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 12:25:28.955015 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.954061    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 12:25:28.955015 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.954930    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 12:25:28.955015 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.954939    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 12:25:28.955015 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.954994    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:25:28.955015 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.954999    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:25:28.955015 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955004    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:25:28.955015 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955008    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:25:28.955015 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955012    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:25:28.955015 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955015    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955018    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955022    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955025    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955029    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955031    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955034    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955037    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955040    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955043    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955047    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955050    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955054    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955057    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955060    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955062    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955065    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955068    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955071    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:25:28.955372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955073    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955076    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955079    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955081    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955084    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955087    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955090    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955093    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955095    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955098    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955101    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955103    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955106    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955108    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955111    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955114    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955117    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955120    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955123    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955125    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:25:28.955846 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955128    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955131    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955133    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955136    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955138    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955141    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955143    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955146    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955148    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955151    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955153    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955156    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955158    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955161    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955163    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955166    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955168    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955171    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955174    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955177    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:25:28.956337 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955180    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955183    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955185    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955188    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955190    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955193    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955195    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955198    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955201    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955204    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955206    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955209    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955211    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955214    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955216    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955219    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955222    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955224    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955226    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955229    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:25:28.956823 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955232    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:25:28.957373 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955234    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:25:28.957373 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955238    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:25:28.957373 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955242    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:25:28.957373 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955244    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:25:28.957373 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955247    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:25:28.957373 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955250    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:25:28.957373 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955252    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:25:28.957373 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955255    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:25:28.957373 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955257    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:25:28.957373 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955260    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:25:28.957373 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955263    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:25:28.957373 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955266    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:25:28.957373 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955269    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:25:28.957373 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955271    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:25:28.957373 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.955276    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955391    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955397    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955400    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955402    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955406    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955410    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955412    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955415    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955418    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955421    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955424    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955427    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955429    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955432    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955435    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955437    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955441    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955445    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955448    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 12:25:28.957757 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955450    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955453    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955456    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955458    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955461    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955463    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955466    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955469    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955472    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955475    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955478    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955481    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955484    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955486    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955489    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955492    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955494    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955497    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955500    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 12:25:28.958265 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955502    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955505    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955507    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955510    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955512    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955515    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955517    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955520    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955522    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955525    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955527    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955530    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955532    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955535    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955537    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955540    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955542    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955545    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955547    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955550    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 12:25:28.958720 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955552    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955555    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955558    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955561    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955564    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955567    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955569    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955572    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955574    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955577    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955580    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955582    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955585    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955588    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955592    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955595    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955598    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955601    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955604    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 12:25:28.959216 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955606    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955609    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955612    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955615    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955617    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955620    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955623    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955626    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955628    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955631    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955633    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955636    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955638    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955641    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955644    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955647    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955650    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955653    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955655    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955659    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 12:25:28.959700 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955662    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 12:25:28.960196 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:28.955665    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 12:25:28.960196 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.955670    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 12:25:28.960196 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.955767    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 12:25:28.960196 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.958118    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 12:25:28.960196 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.959219    2579 server.go:1019] "Starting client certificate rotation"
Sep 15 12:25:28.960196 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.959321    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 12:25:28.960196 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.959354    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 12:25:28.962091 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.962077    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 12:25:28.963222 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.963205    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 12:25:28.974708 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.974688    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 15 12:25:28.978439 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.978424    2579 log.go:25] "Validated CRI v1 image API"
Sep 15 12:25:28.979282 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.979262    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 12:25:28.982602 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.982581    2579 fs.go:135] Filesystem UUIDs: map[025cad81-0f63-40d0-88dd-b3e7af61bdd8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cfdbcb7e-f9ec-4308-9dde-10d14f93d12d:/dev/nvme0n1p4]
Sep 15 12:25:28.982690 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.982600    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 12:25:28.987162 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.987143    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 12:25:28.988895 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.988732    2579 manager.go:216] Machine: {Timestamp:2026-09-15 12:25:28.987398926 +0000 UTC m=+0.372109801 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105807 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e40f6971f82a20f5736e3ee3f916a SystemUUID:ec2e40f6-971f-82a2-0f57-36e3ee3f916a BootID:596bd044-4cde-4ca5-9e0d-591ecb78c649 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c9:5e:80:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c9:5e:80:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:68:09:13:98:22 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 12:25:28.988895 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.988890    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 12:25:28.989066 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.989049    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 12:25:28.992719 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.992698    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 12:25:28.992874 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.992723    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-17.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 12:25:28.993560 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.993550    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 12:25:28.993592 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.993562    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 12:25:28.993592 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.993570    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 12:25:28.994451 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.994440    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 12:25:28.995646 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.995636    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 12:25:28.999964 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:28.999952    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 12:25:29.000082 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.000073    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 12:25:29.000123 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.000101    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 12:25:29.000123 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.000110    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 12:25:29.000123 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.000124    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 12:25:29.000225 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.000133    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 15 12:25:29.000225 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.000141    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 12:25:29.000501 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.000490    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 12:25:29.000537 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.000513    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 12:25:29.004465 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.004441    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 12:25:29.007197 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.007184    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 12:25:29.007254 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.007214    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 12:25:29.008522 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.008504    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6qnnc"
Sep 15 12:25:29.010075 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.010061    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 12:25:29.010139 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.010082    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 12:25:29.010139 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.010091    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 12:25:29.010139 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.010097    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 12:25:29.010139 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.010106    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 12:25:29.010139 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.010113    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 12:25:29.010139 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.010136    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 12:25:29.010305 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.010143    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 12:25:29.010305 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.010141    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-17.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 12:25:29.010305 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.010151    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 12:25:29.010305 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.010164    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 12:25:29.010305 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.010177    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 12:25:29.010305 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.010205    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 12:25:29.010305 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.010213    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 12:25:29.012199 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.012189    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 12:25:29.012244 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.012201    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 12:25:29.016206 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.016191    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6qnnc"
Sep 15 12:25:29.016206 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.016200    2579 server.go:1268] "Started kubelet"
Sep 15 12:25:29.016309 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.016234    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 12:25:29.016309 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.016290    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 12:25:29.020221 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.019790    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 12:25:29.020221 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.020047    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 12:25:29.020248 ip-10-0-129-17 systemd[1]: Started Kubernetes Kubelet.
Sep 15 12:25:29.024471 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.024449    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-17.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 12:25:29.025282 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.025262    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 12:25:29.025371 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.025303    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 12:25:29.031315 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.031298    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 12:25:29.031388 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.031316    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 12:25:29.031455 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.031390    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-17.ec2.internal\" not found"
Sep 15 12:25:29.032051 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.032034    2579 factory.go:55] Registering systemd factory
Sep 15 12:25:29.032132 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.032074    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 15 12:25:29.032187 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.032146    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 12:25:29.032240 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.032218    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 12:25:29.032240 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.032233    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 12:25:29.032334 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.032318    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 12:25:29.032334 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.032333    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 12:25:29.032443 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.032327    2579 factory.go:153] Registering CRI-O factory
Sep 15 12:25:29.032443 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.032355    2579 factory.go:223] Registration of the crio container factory successfully
Sep 15 12:25:29.032443 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.032400    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 12:25:29.032443 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.032419    2579 factory.go:103] Registering Raw factory
Sep 15 12:25:29.032443 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.032431    2579 manager.go:1196] Started watching for new ooms in manager
Sep 15 12:25:29.032748 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.032736    2579 manager.go:349] Starting recovery of all containers
Sep 15 12:25:29.033665 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.033645    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:25:29.036394 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.036370    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-17.ec2.internal\" not found" node="ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.039844 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.039803    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 12:25:29.042636 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.042616    2579 manager.go:354] Recovery completed
Sep 15 12:25:29.046647 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.046635    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:25:29.050380 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.050341    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:25:29.050440 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.050394    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:25:29.050440 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.050405    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:25:29.052558 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.052473    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 12:25:29.052558 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.052487    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:25:29.052558 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.052491    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 12:25:29.052558 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.052510    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 12:25:29.053516 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.053408    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:25:29.053605 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.053528    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:25:29.053605 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.053538    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:25:29.054100 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.054084    2579 policy_none.go:49] "None policy: Start"
Sep 15 12:25:29.054100 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.054101    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 12:25:29.054227 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.054111    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 12:25:29.055455 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.055443    2579 policy_none.go:47] "Start"
Sep 15 12:25:29.095441 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.095420    2579 manager.go:337] "Starting Device Plugin manager"
Sep 15 12:25:29.109979 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.095462    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 12:25:29.109979 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.095475    2579 server.go:87] "Starting device plugin registration server"
Sep 15 12:25:29.109979 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.095779    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 12:25:29.109979 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.095791    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 12:25:29.109979 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.095892    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 12:25:29.109979 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.095962    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 12:25:29.109979 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.095968    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 12:25:29.109979 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.096511    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 12:25:29.109979 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.096564    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-17.ec2.internal\" not found"
Sep 15 12:25:29.109979 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.098760    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 12:25:29.109979 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.098800    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 12:25:29.109979 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.098820    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 12:25:29.109979 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.098884    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 12:25:29.109979 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.100978    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:25:29.196078 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.196010    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:25:29.198077 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.198064    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:25:29.198182 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.198090    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:25:29.198182 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.198102    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:25:29.198182 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.198131    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.199128 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.199113    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-17.ec2.internal"]
Sep 15 12:25:29.199182 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.199173    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:25:29.200935 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.200922    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:25:29.200994 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.200945    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:25:29.200994 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.200957    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:25:29.203038 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.203026    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:25:29.203170 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.203156    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.203203 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.203187    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:25:29.203748 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.203731    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:25:29.203873 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.203751    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:25:29.203873 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.203773    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:25:29.203873 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.203785    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:25:29.203873 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.203756    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:25:29.204069 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.203881    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:25:29.205068 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.205055    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.205111 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.205075    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-17.ec2.internal\": node \"ip-10-0-129-17.ec2.internal\" not found"
Sep 15 12:25:29.205932 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.205917    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.206008 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.205946    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 12:25:29.206634 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.206621    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 12:25:29.206701 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.206646    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 12:25:29.206701 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.206661    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeHasSufficientPID"
Sep 15 12:25:29.221223 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.221202    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-17.ec2.internal\" not found"
Sep 15 12:25:29.222415 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.222401    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-17.ec2.internal\" not found" node="ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.226036 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.226020    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-17.ec2.internal\" not found" node="ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.322052 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.322029    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-17.ec2.internal\" not found"
Sep 15 12:25:29.333607 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.333592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba6038d6a5b6f0c809b6edabd63fb379-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal\" (UID: \"ba6038d6a5b6f0c809b6edabd63fb379\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.333677 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.333614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba6038d6a5b6f0c809b6edabd63fb379-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal\" (UID: \"ba6038d6a5b6f0c809b6edabd63fb379\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.333677 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.333648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0eab95c6c565e847818c125e64a2e65f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-17.ec2.internal\" (UID: \"0eab95c6c565e847818c125e64a2e65f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.422398 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.422371    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-17.ec2.internal\" not found"
Sep 15 12:25:29.434701 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.434683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba6038d6a5b6f0c809b6edabd63fb379-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal\" (UID: \"ba6038d6a5b6f0c809b6edabd63fb379\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.434783 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.434710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba6038d6a5b6f0c809b6edabd63fb379-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal\" (UID: \"ba6038d6a5b6f0c809b6edabd63fb379\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.434783 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.434727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0eab95c6c565e847818c125e64a2e65f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-17.ec2.internal\" (UID: \"0eab95c6c565e847818c125e64a2e65f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.434900 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.434793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0eab95c6c565e847818c125e64a2e65f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-17.ec2.internal\" (UID: \"0eab95c6c565e847818c125e64a2e65f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.434900 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.434796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba6038d6a5b6f0c809b6edabd63fb379-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal\" (UID: \"ba6038d6a5b6f0c809b6edabd63fb379\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.434900 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.434814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba6038d6a5b6f0c809b6edabd63fb379-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal\" (UID: \"ba6038d6a5b6f0c809b6edabd63fb379\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.523162 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.523089    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-17.ec2.internal\" not found"
Sep 15 12:25:29.526253 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.526237    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.529845 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.529814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-17.ec2.internal"
Sep 15 12:25:29.624082 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.624056    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-17.ec2.internal\" not found"
Sep 15 12:25:29.727440 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.727403    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-17.ec2.internal\" not found"
Sep 15 12:25:29.831875 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.831816    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-17.ec2.internal\" not found"
Sep 15 12:25:29.938418 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:29.938381    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-17.ec2.internal\" not found"
Sep 15 12:25:29.959102 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.959070    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 12:25:29.959253 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.959231    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 12:25:29.959299 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:29.959261    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 12:25:30.022294 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:30.022260    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:25:30.025549 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:30.025513    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 12:20:29 +0000 UTC" deadline="2028-04-18 00:49:58.661062514 +0000 UTC"
Sep 15 12:25:30.025624 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:30.025550    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13932h24m28.635516527s"
Sep 15 12:25:30.038520 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:30.038499    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-17.ec2.internal\" not found"
Sep 15 12:25:30.139123 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:30.139063    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-17.ec2.internal\" not found"
Sep 15 12:25:30.240091 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:30.240067    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-17.ec2.internal\" not found"
Sep 15 12:25:30.262943 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:30.262914    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:25:30.336653 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:30.336620    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal"
Sep 15 12:25:30.347986 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:30.347967    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 12:25:30.349330 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:30.349314    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-17.ec2.internal"
Sep 15 12:25:30.358399 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:30.358384    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 12:25:31.003253 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.003223    2579 apiserver.go:52] "Watching apiserver"
Sep 15 12:25:31.012215 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.012187    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 12:25:31.012553 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.012532    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-66lhz","openshift-multus/multus-additional-cni-plugins-ht8cc","openshift-network-diagnostics/network-check-target-n4m28","openshift-ovn-kubernetes/ovnkube-node-rpkkg","kube-system/konnectivity-agent-4pc2d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr","openshift-cluster-node-tuning-operator/tuned-rdsrq","openshift-image-registry/node-ca-blcfn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal","openshift-multus/multus-xbbzp","openshift-multus/network-metrics-daemon-bsb55","openshift-network-operator/iptables-alerter-gvtvg","kube-system/global-pull-secret-syncer-6txlk","kube-system/kube-apiserver-proxy-ip-10-0-129-17.ec2.internal"]
Sep 15 12:25:31.015102 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.015082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.017235 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.017213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:31.017338 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.017275    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:31.017820 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.017803    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 12:25:31.017948 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.017856    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x9pmc\""
Sep 15 12:25:31.018033 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.017948    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 12:25:31.019313 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.019298    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:31.019394 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.019347    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:25:31.021541 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.021527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.023711 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.023686    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4pc2d"
Sep 15 12:25:31.024123 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.024104    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 12:25:31.024329 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.024314    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 12:25:31.024502 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.024487    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 12:25:31.025120 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.025107    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jx5lj\""
Sep 15 12:25:31.025214 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.025132    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 12:25:31.025214 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.025138    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 12:25:31.025214 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.025139    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 12:25:31.025884 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.025862    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 12:25:31.025884 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.025879    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 12:25:31.026043 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.026000    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hkhnq\""
Sep 15 12:25:31.026194 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.026177    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.028267 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.028250    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 12:25:31.028535 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.028519    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cqcf2\""
Sep 15 12:25:31.028628 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.028539    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 12:25:31.028628 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.028520    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 12:25:31.028628 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.028605    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-blcfn"
Sep 15 12:25:31.030646 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.030632    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 12:25:31.030876 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.030855    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 12:25:31.030876 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.030876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fwljz\""
Sep 15 12:25:31.031064 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.030916    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 12:25:31.031064 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.030982    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-66lhz"
Sep 15 12:25:31.031959 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.031942    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 12:25:31.033043 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.033029    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 12:25:31.033283 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.033270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.033416 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.033403    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tzzm6\""
Sep 15 12:25:31.033963 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.033949    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 12:25:31.035736 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.035720    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 12:25:31.035839 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.035819    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.036595 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.036574    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 12:25:31.037337 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.037000    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 12:25:31.037337 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.037050    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 12:25:31.037497 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.037440    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2bb7x\""
Sep 15 12:25:31.037557 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.037542    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 12:25:31.039073 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.038348    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 12:25:31.039250 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.039230    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pwcpq\""
Sep 15 12:25:31.040505 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.040484    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:31.040602 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.040561    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:25:31.042781 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.042765    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gvtvg"
Sep 15 12:25:31.044120 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.044104    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 12:25:31.044892 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.044876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5w4p\" (UniqueName: \"kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p\") pod \"network-check-target-n4m28\" (UID: \"2b715742-769a-4263-a81d-afd9e2c29183\") " pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:31.044967 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.044899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-run-openvswitch\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.044967 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.044940    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c05cc62-0238-4991-96af-7633ec6f0e05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.045058 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.044983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/594efe57-d61c-4858-8df4-41ebf3957073-etc-tuned\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.045058 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-node-log\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.045136 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-cni-netd\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.045136 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045029    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 12:25:31.045136 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-ovnkube-config\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.045136 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-ovn-node-metrics-cert\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.045311 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e17c9098-956f-4d77-a535-5b09749f3a95-konnectivity-ca\") pod \"konnectivity-agent-4pc2d\" (UID: \"e17c9098-956f-4d77-a535-5b09749f3a95\") " pod="kube-system/konnectivity-agent-4pc2d"
Sep 15 12:25:31.045311 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045155    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-kubernetes\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.045311 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzscb\" (UniqueName: \"kubernetes.io/projected/594efe57-d61c-4858-8df4-41ebf3957073-kube-api-access-hzscb\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.045311 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-registration-dir\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.045311 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-var-lib-openvswitch\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.045311 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-sysctl-d\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.045311 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045255    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-sysctl-conf\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.045311 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-systemd\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.045311 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-socket-dir\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.045311 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045305    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 12:25:31.045709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-device-dir\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.045709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-sys-fs\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.045709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:31.045709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-527l7\" (UniqueName: \"kubernetes.io/projected/6c05cc62-0238-4991-96af-7633ec6f0e05-kube-api-access-527l7\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.045709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-etc-selinux\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.045709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.045709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-kubelet-config\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:31.045709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-kubelet\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.045709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045547    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 12:25:31.045709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045547    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v52jg\""
Sep 15 12:25:31.045709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j9lq\" (UniqueName: \"kubernetes.io/projected/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-kube-api-access-4j9lq\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.045709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7xt9\" (UniqueName: \"kubernetes.io/projected/866085d7-f0ed-4104-bd66-d5964b6619c9-kube-api-access-z7xt9\") pod \"node-resolver-66lhz\" (UID: \"866085d7-f0ed-4104-bd66-d5964b6619c9\") " pod="openshift-dns/node-resolver-66lhz"
Sep 15 12:25:31.045709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-modprobe-d\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.045709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-systemd-units\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.045709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-etc-openvswitch\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-run-ovn-kubernetes\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866085d7-f0ed-4104-bd66-d5964b6619c9-tmp-dir\") pod \"node-resolver-66lhz\" (UID: \"866085d7-f0ed-4104-bd66-d5964b6619c9\") " pod="openshift-dns/node-resolver-66lhz"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c05cc62-0238-4991-96af-7633ec6f0e05-cni-binary-copy\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-sys\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-run\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrn4m\" (UniqueName: \"kubernetes.io/projected/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-kube-api-access-jrn4m\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-slash\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-run-systemd\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc-host\") pod \"node-ca-blcfn\" (UID: \"1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc\") " pod="openshift-image-registry/node-ca-blcfn"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.045965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-run-ovn\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-cni-bin\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/866085d7-f0ed-4104-bd66-d5964b6619c9-hosts-file\") pod \"node-resolver-66lhz\" (UID: \"866085d7-f0ed-4104-bd66-d5964b6619c9\") " pod="openshift-dns/node-resolver-66lhz"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c05cc62-0238-4991-96af-7633ec6f0e05-cnibin\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c05cc62-0238-4991-96af-7633ec6f0e05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-host\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.046421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.046927 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwsxh\" (UniqueName: \"kubernetes.io/projected/1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc-kube-api-access-lwsxh\") pod \"node-ca-blcfn\" (UID: \"1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc\") " pod="openshift-image-registry/node-ca-blcfn"
Sep 15 12:25:31.046927 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/594efe57-d61c-4858-8df4-41ebf3957073-tmp\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.046927 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc-serviceca\") pod \"node-ca-blcfn\" (UID: \"1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc\") " pod="openshift-image-registry/node-ca-blcfn"
Sep 15 12:25:31.046927 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-ovnkube-script-lib\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.046927 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-log-socket\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.046927 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-env-overrides\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.046927 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-run-netns\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.046927 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e17c9098-956f-4d77-a535-5b09749f3a95-agent-certs\") pod \"konnectivity-agent-4pc2d\" (UID: \"e17c9098-956f-4d77-a535-5b09749f3a95\") " pod="kube-system/konnectivity-agent-4pc2d"
Sep 15 12:25:31.046927 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c05cc62-0238-4991-96af-7633ec6f0e05-system-cni-dir\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.046927 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c05cc62-0238-4991-96af-7633ec6f0e05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.046927 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-var-lib-kubelet\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.046927 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-dbus\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:31.046927 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-sysconfig\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.046927 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.046529    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-lib-modules\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.068423 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.068395    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bqkwz"
Sep 15 12:25:31.076659 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.076641    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bqkwz"
Sep 15 12:25:31.132957 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.132940    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 12:25:31.147329 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.147310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.147407 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.147341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs\") pod \"network-metrics-daemon-bsb55\" (UID: \"15a5076c-b542-4a0d-bad7-cc06b81bbe3e\") " pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:31.147407 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.147382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-multus-socket-dir-parent\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.147407 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.147396    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.147515 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.147435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-run-multus-certs\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.147515 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.147473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-kubelet-config\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:31.147690 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.147671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-kubelet-config\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:31.147958 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.147943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-kubelet\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.148008 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.147966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j9lq\" (UniqueName: \"kubernetes.io/projected/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-kube-api-access-4j9lq\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.148008 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.147981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7xt9\" (UniqueName: \"kubernetes.io/projected/866085d7-f0ed-4104-bd66-d5964b6619c9-kube-api-access-z7xt9\") pod \"node-resolver-66lhz\" (UID: \"866085d7-f0ed-4104-bd66-d5964b6619c9\") " pod="openshift-dns/node-resolver-66lhz"
Sep 15 12:25:31.148008 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.147997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-modprobe-d\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.148159 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.148015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lch8t\" (UniqueName: \"kubernetes.io/projected/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-kube-api-access-lch8t\") pod \"network-metrics-daemon-bsb55\" (UID: \"15a5076c-b542-4a0d-bad7-cc06b81bbe3e\") " pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:31.148159 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.148037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-systemd-units\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.148159 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.148056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-kubelet\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.148159 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.148059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-etc-openvswitch\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.148159 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.148104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-etc-openvswitch\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.148159 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.148091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-systemd-units\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.148378 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.148124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-run-ovn-kubernetes\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.148447 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.148300    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-modprobe-d\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.148602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-run-ovn-kubernetes\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.148668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866085d7-f0ed-4104-bd66-d5964b6619c9-tmp-dir\") pod \"node-resolver-66lhz\" (UID: \"866085d7-f0ed-4104-bd66-d5964b6619c9\") " pod="openshift-dns/node-resolver-66lhz"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.148771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c05cc62-0238-4991-96af-7633ec6f0e05-cni-binary-copy\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.148802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-sys\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.149382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c05cc62-0238-4991-96af-7633ec6f0e05-cni-binary-copy\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.149470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-run-k8s-cni-cncf-io\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.149511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-run\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.149575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrn4m\" (UniqueName: \"kubernetes.io/projected/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-kube-api-access-jrn4m\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.149606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-hostroot\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.149640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4kpg\" (UniqueName: \"kubernetes.io/projected/6fe7f374-634a-4074-97b1-21a380d41caa-kube-api-access-t4kpg\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.149675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-slash\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.149705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-run-systemd\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.149735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-run-netns\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.149761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-var-lib-kubelet\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.149841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7x4g\" (UniqueName: \"kubernetes.io/projected/0c0fc181-a9d3-4e90-ae03-90b1d11d4146-kube-api-access-l7x4g\") pod \"iptables-alerter-gvtvg\" (UID: \"0c0fc181-a9d3-4e90-ae03-90b1d11d4146\") " pod="openshift-network-operator/iptables-alerter-gvtvg"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.149872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc-host\") pod \"node-ca-blcfn\" (UID: \"1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc\") " pod="openshift-image-registry/node-ca-blcfn"
Sep 15 12:25:31.150803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.149902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fe7f374-634a-4074-97b1-21a380d41caa-cni-binary-copy\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.149959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6fe7f374-634a-4074-97b1-21a380d41caa-multus-daemon-config\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.149990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-etc-kubernetes\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-run-ovn\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-cni-bin\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/866085d7-f0ed-4104-bd66-d5964b6619c9-hosts-file\") pod \"node-resolver-66lhz\" (UID: \"866085d7-f0ed-4104-bd66-d5964b6619c9\") " pod="openshift-dns/node-resolver-66lhz"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c05cc62-0238-4991-96af-7633ec6f0e05-cnibin\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c05cc62-0238-4991-96af-7633ec6f0e05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150277    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c05cc62-0238-4991-96af-7633ec6f0e05-cnibin\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-host\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwsxh\" (UniqueName: \"kubernetes.io/projected/1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc-kube-api-access-lwsxh\") pod \"node-ca-blcfn\" (UID: \"1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc\") " pod="openshift-image-registry/node-ca-blcfn"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/594efe57-d61c-4858-8df4-41ebf3957073-tmp\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc-serviceca\") pod \"node-ca-blcfn\" (UID: \"1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc\") " pod="openshift-image-registry/node-ca-blcfn"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-cnibin\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-var-lib-cni-bin\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-var-lib-cni-multus\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.151667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-ovnkube-script-lib\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-log-socket\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-env-overrides\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-multus-cni-dir\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0c0fc181-a9d3-4e90-ae03-90b1d11d4146-iptables-alerter-script\") pod \"iptables-alerter-gvtvg\" (UID: \"0c0fc181-a9d3-4e90-ae03-90b1d11d4146\") " pod="openshift-network-operator/iptables-alerter-gvtvg"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/866085d7-f0ed-4104-bd66-d5964b6619c9-hosts-file\") pod \"node-resolver-66lhz\" (UID: \"866085d7-f0ed-4104-bd66-d5964b6619c9\") " pod="openshift-dns/node-resolver-66lhz"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-run-systemd\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-run-netns\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-sys\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150917    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-run-ovn\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-slash\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-cni-bin\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.150994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-run\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.151153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-host\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.151199    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc-host\") pod \"node-ca-blcfn\" (UID: \"1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc\") " pod="openshift-image-registry/node-ca-blcfn"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.151320    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/866085d7-f0ed-4104-bd66-d5964b6619c9-tmp-dir\") pod \"node-resolver-66lhz\" (UID: \"866085d7-f0ed-4104-bd66-d5964b6619c9\") " pod="openshift-dns/node-resolver-66lhz"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.151329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c05cc62-0238-4991-96af-7633ec6f0e05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.151406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.152352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.151408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e17c9098-956f-4d77-a535-5b09749f3a95-agent-certs\") pod \"konnectivity-agent-4pc2d\" (UID: \"e17c9098-956f-4d77-a535-5b09749f3a95\") " pod="kube-system/konnectivity-agent-4pc2d"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.151429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc-serviceca\") pod \"node-ca-blcfn\" (UID: \"1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc\") " pod="openshift-image-registry/node-ca-blcfn"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.151443    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.151479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-run-netns\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.151554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c05cc62-0238-4991-96af-7633ec6f0e05-system-cni-dir\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.151607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-log-socket\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.151718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c05cc62-0238-4991-96af-7633ec6f0e05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.151755    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c05cc62-0238-4991-96af-7633ec6f0e05-system-cni-dir\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.151764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-env-overrides\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.151768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c0fc181-a9d3-4e90-ae03-90b1d11d4146-host-slash\") pod \"iptables-alerter-gvtvg\" (UID: \"0c0fc181-a9d3-4e90-ae03-90b1d11d4146\") " pod="openshift-network-operator/iptables-alerter-gvtvg"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-var-lib-kubelet\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-dbus\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-sysconfig\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c05cc62-0238-4991-96af-7633ec6f0e05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-lib-modules\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-multus-conf-dir\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-ovnkube-script-lib\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.153335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5w4p\" (UniqueName: \"kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p\") pod \"network-check-target-n4m28\" (UID: \"2b715742-769a-4263-a81d-afd9e2c29183\") " pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-dbus\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-lib-modules\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-run-openvswitch\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c05cc62-0238-4991-96af-7633ec6f0e05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/594efe57-d61c-4858-8df4-41ebf3957073-etc-tuned\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-sysconfig\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-run-openvswitch\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-var-lib-kubelet\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-node-log\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-node-log\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-cni-netd\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-ovnkube-config\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-ovn-node-metrics-cert\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-host-cni-netd\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e17c9098-956f-4d77-a535-5b09749f3a95-konnectivity-ca\") pod \"konnectivity-agent-4pc2d\" (UID: \"e17c9098-956f-4d77-a535-5b09749f3a95\") " pod="kube-system/konnectivity-agent-4pc2d"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152949    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-kubernetes\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.154290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.152977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzscb\" (UniqueName: \"kubernetes.io/projected/594efe57-d61c-4858-8df4-41ebf3957073-kube-api-access-hzscb\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-registration-dir\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-var-lib-openvswitch\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-sysctl-d\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153085    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-sysctl-conf\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-systemd\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-socket-dir\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-device-dir\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-sys-fs\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-system-cni-dir\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c05cc62-0238-4991-96af-7633ec6f0e05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-527l7\" (UniqueName: \"kubernetes.io/projected/6c05cc62-0238-4991-96af-7633ec6f0e05-kube-api-access-527l7\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-etc-selinux\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e17c9098-956f-4d77-a535-5b09749f3a95-konnectivity-ca\") pod \"konnectivity-agent-4pc2d\" (UID: \"e17c9098-956f-4d77-a535-5b09749f3a95\") " pod="kube-system/konnectivity-agent-4pc2d"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-etc-selinux\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.153760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-kubernetes\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.155269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.154020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-registration-dir\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.156105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.154062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-var-lib-openvswitch\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.156105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.154089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-ovnkube-config\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.156105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.154155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-sysctl-d\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.156105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.154250    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-sysctl-conf\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.156105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.154294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/594efe57-d61c-4858-8df4-41ebf3957073-etc-systemd\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.156105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.154419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-socket-dir\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.156105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.154472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-device-dir\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.156105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.154519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-sys-fs\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.156105 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.154602    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:25:31.156105 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.154702    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret podName:f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:31.654664396 +0000 UTC m=+3.039375285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret") pod "global-pull-secret-syncer-6txlk" (UID: "f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:25:31.156105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.155237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/594efe57-d61c-4858-8df4-41ebf3957073-tmp\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.156105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.155362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/594efe57-d61c-4858-8df4-41ebf3957073-etc-tuned\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.156105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.155757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-ovn-node-metrics-cert\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.156782 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.156318    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e17c9098-956f-4d77-a535-5b09749f3a95-agent-certs\") pod \"konnectivity-agent-4pc2d\" (UID: \"e17c9098-956f-4d77-a535-5b09749f3a95\") " pod="kube-system/konnectivity-agent-4pc2d"
Sep 15 12:25:31.157077 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.157052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7xt9\" (UniqueName: \"kubernetes.io/projected/866085d7-f0ed-4104-bd66-d5964b6619c9-kube-api-access-z7xt9\") pod \"node-resolver-66lhz\" (UID: \"866085d7-f0ed-4104-bd66-d5964b6619c9\") " pod="openshift-dns/node-resolver-66lhz"
Sep 15 12:25:31.157937 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.157901    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j9lq\" (UniqueName: \"kubernetes.io/projected/371b2ab6-5733-4e72-80bf-36ddbdc2ea74-kube-api-access-4j9lq\") pod \"ovnkube-node-rpkkg\" (UID: \"371b2ab6-5733-4e72-80bf-36ddbdc2ea74\") " pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.158195 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.158174    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:25:31.158275 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.158198    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:25:31.158275 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.158212    2579 projected.go:196] Error preparing data for projected volume kube-api-access-q5w4p for pod openshift-network-diagnostics/network-check-target-n4m28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:25:31.158275 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.158256    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p podName:2b715742-769a-4263-a81d-afd9e2c29183 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:31.658244863 +0000 UTC m=+3.042955727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q5w4p" (UniqueName: "kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p") pod "network-check-target-n4m28" (UID: "2b715742-769a-4263-a81d-afd9e2c29183") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:25:31.158471 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.158333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrn4m\" (UniqueName: \"kubernetes.io/projected/a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139-kube-api-access-jrn4m\") pod \"aws-ebs-csi-driver-node-fcrcr\" (UID: \"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.158519 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.158478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwsxh\" (UniqueName: \"kubernetes.io/projected/1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc-kube-api-access-lwsxh\") pod \"node-ca-blcfn\" (UID: \"1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc\") " pod="openshift-image-registry/node-ca-blcfn"
Sep 15 12:25:31.160583 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.160563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-527l7\" (UniqueName: \"kubernetes.io/projected/6c05cc62-0238-4991-96af-7633ec6f0e05-kube-api-access-527l7\") pod \"multus-additional-cni-plugins-ht8cc\" (UID: \"6c05cc62-0238-4991-96af-7633ec6f0e05\") " pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.161127 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.161108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzscb\" (UniqueName: \"kubernetes.io/projected/594efe57-d61c-4858-8df4-41ebf3957073-kube-api-access-hzscb\") pod \"tuned-rdsrq\" (UID: \"594efe57-d61c-4858-8df4-41ebf3957073\") " pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.254105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c0fc181-a9d3-4e90-ae03-90b1d11d4146-host-slash\") pod \"iptables-alerter-gvtvg\" (UID: \"0c0fc181-a9d3-4e90-ae03-90b1d11d4146\") " pod="openshift-network-operator/iptables-alerter-gvtvg"
Sep 15 12:25:31.254105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-multus-conf-dir\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254235 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c0fc181-a9d3-4e90-ae03-90b1d11d4146-host-slash\") pod \"iptables-alerter-gvtvg\" (UID: \"0c0fc181-a9d3-4e90-ae03-90b1d11d4146\") " pod="openshift-network-operator/iptables-alerter-gvtvg"
Sep 15 12:25:31.254235 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-multus-conf-dir\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254235 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-system-cni-dir\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254357 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs\") pod \"network-metrics-daemon-bsb55\" (UID: \"15a5076c-b542-4a0d-bad7-cc06b81bbe3e\") " pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:31.254357 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-multus-socket-dir-parent\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254357 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-run-multus-certs\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254357 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lch8t\" (UniqueName: \"kubernetes.io/projected/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-kube-api-access-lch8t\") pod \"network-metrics-daemon-bsb55\" (UID: \"15a5076c-b542-4a0d-bad7-cc06b81bbe3e\") " pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:31.254357 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-system-cni-dir\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254357 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254336    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-run-multus-certs\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-run-k8s-cni-cncf-io\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254380    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-multus-socket-dir-parent\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.254389    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-hostroot\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-run-k8s-cni-cncf-io\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.254444    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs podName:15a5076c-b542-4a0d-bad7-cc06b81bbe3e nodeName:}" failed. No retries permitted until 2026-09-15 12:25:31.754426482 +0000 UTC m=+3.139137346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs") pod "network-metrics-daemon-bsb55" (UID: "15a5076c-b542-4a0d-bad7-cc06b81bbe3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-hostroot\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4kpg\" (UniqueName: \"kubernetes.io/projected/6fe7f374-634a-4074-97b1-21a380d41caa-kube-api-access-t4kpg\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-run-netns\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-var-lib-kubelet\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-var-lib-kubelet\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254579    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-run-netns\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7x4g\" (UniqueName: \"kubernetes.io/projected/0c0fc181-a9d3-4e90-ae03-90b1d11d4146-kube-api-access-l7x4g\") pod \"iptables-alerter-gvtvg\" (UID: \"0c0fc181-a9d3-4e90-ae03-90b1d11d4146\") " pod="openshift-network-operator/iptables-alerter-gvtvg"
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fe7f374-634a-4074-97b1-21a380d41caa-cni-binary-copy\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6fe7f374-634a-4074-97b1-21a380d41caa-multus-daemon-config\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.254660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-etc-kubernetes\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.255429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-cnibin\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.255429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254721    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-var-lib-cni-bin\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.255429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-var-lib-cni-multus\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.255429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-multus-cni-dir\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.255429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254783    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-etc-kubernetes\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.255429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-cnibin\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.255429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-var-lib-cni-multus\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.255429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0c0fc181-a9d3-4e90-ae03-90b1d11d4146-iptables-alerter-script\") pod \"iptables-alerter-gvtvg\" (UID: \"0c0fc181-a9d3-4e90-ae03-90b1d11d4146\") " pod="openshift-network-operator/iptables-alerter-gvtvg"
Sep 15 12:25:31.255429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254872    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-multus-cni-dir\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.255429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.254924    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6fe7f374-634a-4074-97b1-21a380d41caa-host-var-lib-cni-bin\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.255429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.255139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6fe7f374-634a-4074-97b1-21a380d41caa-cni-binary-copy\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.255429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.255264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6fe7f374-634a-4074-97b1-21a380d41caa-multus-daemon-config\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.255429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.255323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0c0fc181-a9d3-4e90-ae03-90b1d11d4146-iptables-alerter-script\") pod \"iptables-alerter-gvtvg\" (UID: \"0c0fc181-a9d3-4e90-ae03-90b1d11d4146\") " pod="openshift-network-operator/iptables-alerter-gvtvg"
Sep 15 12:25:31.263041 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.263023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7x4g\" (UniqueName: \"kubernetes.io/projected/0c0fc181-a9d3-4e90-ae03-90b1d11d4146-kube-api-access-l7x4g\") pod \"iptables-alerter-gvtvg\" (UID: \"0c0fc181-a9d3-4e90-ae03-90b1d11d4146\") " pod="openshift-network-operator/iptables-alerter-gvtvg"
Sep 15 12:25:31.263195 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.263173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4kpg\" (UniqueName: \"kubernetes.io/projected/6fe7f374-634a-4074-97b1-21a380d41caa-kube-api-access-t4kpg\") pod \"multus-xbbzp\" (UID: \"6fe7f374-634a-4074-97b1-21a380d41caa\") " pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.263291 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.263276    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lch8t\" (UniqueName: \"kubernetes.io/projected/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-kube-api-access-lch8t\") pod \"network-metrics-daemon-bsb55\" (UID: \"15a5076c-b542-4a0d-bad7-cc06b81bbe3e\") " pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:31.352286 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.352259    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rdsrq"
Sep 15 12:25:31.366891 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.366870    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:25:31.370083 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.370067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:31.387260 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.387237    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 12:25:31.390592 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.390579    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4pc2d"
Sep 15 12:25:31.392919 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.392902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr"
Sep 15 12:25:31.413922 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.413884    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-blcfn"
Sep 15 12:25:31.427366 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.427348    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-66lhz"
Sep 15 12:25:31.439449 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.439432    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ht8cc"
Sep 15 12:25:31.456719 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.456698    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xbbzp"
Sep 15 12:25:31.473632 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.473608    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gvtvg"
Sep 15 12:25:31.659451 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.659364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5w4p\" (UniqueName: \"kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p\") pod \"network-check-target-n4m28\" (UID: \"2b715742-769a-4263-a81d-afd9e2c29183\") " pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:31.659618 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.659512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:31.659618 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.659547    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:25:31.659618 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.659569    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:25:31.659618 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.659583    2579 projected.go:196] Error preparing data for projected volume kube-api-access-q5w4p for pod openshift-network-diagnostics/network-check-target-n4m28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:25:31.659618 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.659609    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:25:31.659810 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.659667    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret podName:f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:32.659652435 +0000 UTC m=+4.044363299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret") pod "global-pull-secret-syncer-6txlk" (UID: "f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:25:31.659810 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.659682    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p podName:2b715742-769a-4263-a81d-afd9e2c29183 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:32.659675273 +0000 UTC m=+4.044386155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q5w4p" (UniqueName: "kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p") pod "network-check-target-n4m28" (UID: "2b715742-769a-4263-a81d-afd9e2c29183") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:25:31.760387 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:31.760353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs\") pod \"network-metrics-daemon-bsb55\" (UID: \"15a5076c-b542-4a0d-bad7-cc06b81bbe3e\") " pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:31.760519 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.760478    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:25:31.760569 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:31.760541    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs podName:15a5076c-b542-4a0d-bad7-cc06b81bbe3e nodeName:}" failed. No retries permitted until 2026-09-15 12:25:32.760528051 +0000 UTC m=+4.145238916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs") pod "network-metrics-daemon-bsb55" (UID: "15a5076c-b542-4a0d-bad7-cc06b81bbe3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:25:32.003588 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:32.003544    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594efe57_d61c_4858_8df4_41ebf3957073.slice/crio-5541a4a4372d9088dbc47773201e2f01849e925219827c5f7ae755620cac8281 WatchSource:0}: Error finding container 5541a4a4372d9088dbc47773201e2f01849e925219827c5f7ae755620cac8281: Status 404 returned error can't find the container with id 5541a4a4372d9088dbc47773201e2f01849e925219827c5f7ae755620cac8281
Sep 15 12:25:32.004747 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:32.004718    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0fc181_a9d3_4e90_ae03_90b1d11d4146.slice/crio-7c94550953ec4d65c336d258e7dfb8f26136e38d7079220f1405652e12b1ac97 WatchSource:0}: Error finding container 7c94550953ec4d65c336d258e7dfb8f26136e38d7079220f1405652e12b1ac97: Status 404 returned error can't find the container with id 7c94550953ec4d65c336d258e7dfb8f26136e38d7079220f1405652e12b1ac97
Sep 15 12:25:32.007042 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:32.007017    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c05cc62_0238_4991_96af_7633ec6f0e05.slice/crio-f77dc5c47cbd1b2826a5bcc2f3891f9967889692b5bd3d03d83741465fdaac31 WatchSource:0}: Error finding container f77dc5c47cbd1b2826a5bcc2f3891f9967889692b5bd3d03d83741465fdaac31: Status 404 returned error can't find the container with id f77dc5c47cbd1b2826a5bcc2f3891f9967889692b5bd3d03d83741465fdaac31
Sep 15 12:25:32.009376 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:32.009357    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:25:32.010753 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:32.010729    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17c9098_956f_4d77_a535_5b09749f3a95.slice/crio-70a582b8673bc581c3f96046cd73c0758f37e0895ef8a34aba99d6cc70380e50 WatchSource:0}: Error finding container 70a582b8673bc581c3f96046cd73c0758f37e0895ef8a34aba99d6cc70380e50: Status 404 returned error can't find the container with id 70a582b8673bc581c3f96046cd73c0758f37e0895ef8a34aba99d6cc70380e50
Sep 15 12:25:32.012995 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:32.012975    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe7f374_634a_4074_97b1_21a380d41caa.slice/crio-84f22b81939835d9488edb3099d51dc12efe8938136f53ee7aff2f3012ee70fa WatchSource:0}: Error finding container 84f22b81939835d9488edb3099d51dc12efe8938136f53ee7aff2f3012ee70fa: Status 404 returned error can't find the container with id 84f22b81939835d9488edb3099d51dc12efe8938136f53ee7aff2f3012ee70fa
Sep 15 12:25:32.051884 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:32.051832    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c61e9c1_fd93_4f6b_ac81_1ac9273d1ecc.slice/crio-9493d14ab5b905cc8024869527858f508e3ac69a936fdc4e2faed602a93ab58f WatchSource:0}: Error finding container 9493d14ab5b905cc8024869527858f508e3ac69a936fdc4e2faed602a93ab58f: Status 404 returned error can't find the container with id 9493d14ab5b905cc8024869527858f508e3ac69a936fdc4e2faed602a93ab58f
Sep 15 12:25:32.079820 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:32.079793    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 12:20:31 +0000 UTC" deadline="2028-03-08 05:02:32.531284135 +0000 UTC"
Sep 15 12:25:32.079820 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:32.079819    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12952h37m0.451468159s"
Sep 15 12:25:32.080125 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:32.080104    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371b2ab6_5733_4e72_80bf_36ddbdc2ea74.slice/crio-a8f062641b77ff8581dcff21c7e79ea08445d126529da10ece17b4395731f9b9 WatchSource:0}: Error finding container a8f062641b77ff8581dcff21c7e79ea08445d126529da10ece17b4395731f9b9: Status 404 returned error can't find the container with id a8f062641b77ff8581dcff21c7e79ea08445d126529da10ece17b4395731f9b9
Sep 15 12:25:32.099807 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:32.099787    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:32.099938 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:32.099916    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:32.103284 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:32.103260    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866085d7_f0ed_4104_bd66_d5964b6619c9.slice/crio-d64f6fc80ce9b82f62158085c858b162b211a69eda112e328535d46773d3b8d2 WatchSource:0}: Error finding container d64f6fc80ce9b82f62158085c858b162b211a69eda112e328535d46773d3b8d2: Status 404 returned error can't find the container with id d64f6fc80ce9b82f62158085c858b162b211a69eda112e328535d46773d3b8d2
Sep 15 12:25:32.104247 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:32.104204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg" event={"ID":"371b2ab6-5733-4e72-80bf-36ddbdc2ea74","Type":"ContainerStarted","Data":"a8f062641b77ff8581dcff21c7e79ea08445d126529da10ece17b4395731f9b9"}
Sep 15 12:25:32.105390 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:32.105367    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-blcfn" event={"ID":"1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc","Type":"ContainerStarted","Data":"9493d14ab5b905cc8024869527858f508e3ac69a936fdc4e2faed602a93ab58f"}
Sep 15 12:25:32.106546 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:32.106473    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xbbzp" event={"ID":"6fe7f374-634a-4074-97b1-21a380d41caa","Type":"ContainerStarted","Data":"84f22b81939835d9488edb3099d51dc12efe8938136f53ee7aff2f3012ee70fa"}
Sep 15 12:25:32.107562 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:32.107544    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4pc2d" event={"ID":"e17c9098-956f-4d77-a535-5b09749f3a95","Type":"ContainerStarted","Data":"70a582b8673bc581c3f96046cd73c0758f37e0895ef8a34aba99d6cc70380e50"}
Sep 15 12:25:32.108521 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:32.108501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ht8cc" event={"ID":"6c05cc62-0238-4991-96af-7633ec6f0e05","Type":"ContainerStarted","Data":"f77dc5c47cbd1b2826a5bcc2f3891f9967889692b5bd3d03d83741465fdaac31"}
Sep 15 12:25:32.109360 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:32.109340    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gvtvg" event={"ID":"0c0fc181-a9d3-4e90-ae03-90b1d11d4146","Type":"ContainerStarted","Data":"7c94550953ec4d65c336d258e7dfb8f26136e38d7079220f1405652e12b1ac97"}
Sep 15 12:25:32.110376 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:32.110358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rdsrq" event={"ID":"594efe57-d61c-4858-8df4-41ebf3957073","Type":"ContainerStarted","Data":"5541a4a4372d9088dbc47773201e2f01849e925219827c5f7ae755620cac8281"}
Sep 15 12:25:32.670889 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:32.670826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5w4p\" (UniqueName: \"kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p\") pod \"network-check-target-n4m28\" (UID: \"2b715742-769a-4263-a81d-afd9e2c29183\") " pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:32.671105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:32.670923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:32.671181 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:32.671119    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:25:32.671313 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:32.671187    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret podName:f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:34.671168023 +0000 UTC m=+6.055878902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret") pod "global-pull-secret-syncer-6txlk" (UID: "f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:25:32.671686 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:32.671662    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:25:32.671778 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:32.671696    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:25:32.671778 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:32.671711    2579 projected.go:196] Error preparing data for projected volume kube-api-access-q5w4p for pod openshift-network-diagnostics/network-check-target-n4m28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:25:32.671893 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:32.671789    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p podName:2b715742-769a-4263-a81d-afd9e2c29183 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:34.671772945 +0000 UTC m=+6.056483824 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q5w4p" (UniqueName: "kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p") pod "network-check-target-n4m28" (UID: "2b715742-769a-4263-a81d-afd9e2c29183") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:25:32.771558 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:32.771516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs\") pod \"network-metrics-daemon-bsb55\" (UID: \"15a5076c-b542-4a0d-bad7-cc06b81bbe3e\") " pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:32.771792 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:32.771684    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:25:32.771792 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:32.771754    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs podName:15a5076c-b542-4a0d-bad7-cc06b81bbe3e nodeName:}" failed. No retries permitted until 2026-09-15 12:25:34.77173332 +0000 UTC m=+6.156444202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs") pod "network-metrics-daemon-bsb55" (UID: "15a5076c-b542-4a0d-bad7-cc06b81bbe3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:25:33.081043 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:33.080921    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 12:20:31 +0000 UTC" deadline="2028-03-09 06:25:12.544039818 +0000 UTC"
Sep 15 12:25:33.081043 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:33.080992    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12977h59m39.463052838s"
Sep 15 12:25:33.102240 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:33.101523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:33.102240 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:33.101648    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:25:33.102240 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:33.102061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:33.102240 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:33.102185    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:25:33.118849 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:33.118809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-66lhz" event={"ID":"866085d7-f0ed-4104-bd66-d5964b6619c9","Type":"ContainerStarted","Data":"d64f6fc80ce9b82f62158085c858b162b211a69eda112e328535d46773d3b8d2"}
Sep 15 12:25:33.576777 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:25:33.576738    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6038d6a5b6f0c809b6edabd63fb379.slice/crio-3ef6c1d67344bce1d95af805c84eb29beb0611f34a188c5e5f7b3275cf1b8d1f WatchSource:0}: Error finding container 3ef6c1d67344bce1d95af805c84eb29beb0611f34a188c5e5f7b3275cf1b8d1f: Status 404 returned error can't find the container with id 3ef6c1d67344bce1d95af805c84eb29beb0611f34a188c5e5f7b3275cf1b8d1f
Sep 15 12:25:34.099617 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:34.099315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:34.100060 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:34.099699    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:34.132833 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:34.132789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal" event={"ID":"ba6038d6a5b6f0c809b6edabd63fb379","Type":"ContainerStarted","Data":"3ef6c1d67344bce1d95af805c84eb29beb0611f34a188c5e5f7b3275cf1b8d1f"}
Sep 15 12:25:34.144808 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:34.144771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-17.ec2.internal" event={"ID":"0eab95c6c565e847818c125e64a2e65f","Type":"ContainerStarted","Data":"3e421a26add60848785143aa9c2e2cc0e59cfce8f3ded0804a38c11d386af98b"}
Sep 15 12:25:34.687250 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:34.687205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5w4p\" (UniqueName: \"kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p\") pod \"network-check-target-n4m28\" (UID: \"2b715742-769a-4263-a81d-afd9e2c29183\") " pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:34.687454 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:34.687277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:34.687454 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:34.687449    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:25:34.687574 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:34.687534    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret podName:f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:38.687512263 +0000 UTC m=+10.072223141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret") pod "global-pull-secret-syncer-6txlk" (UID: "f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:25:34.688030 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:34.688005    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:25:34.688165 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:34.688035    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:25:34.688165 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:34.688049    2579 projected.go:196] Error preparing data for projected volume kube-api-access-q5w4p for pod openshift-network-diagnostics/network-check-target-n4m28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:25:34.688165 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:34.688110    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p podName:2b715742-769a-4263-a81d-afd9e2c29183 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:38.688093815 +0000 UTC m=+10.072804697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q5w4p" (UniqueName: "kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p") pod "network-check-target-n4m28" (UID: "2b715742-769a-4263-a81d-afd9e2c29183") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:25:34.788086 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:34.787968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs\") pod \"network-metrics-daemon-bsb55\" (UID: \"15a5076c-b542-4a0d-bad7-cc06b81bbe3e\") " pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:34.788304 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:34.788141    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:25:34.788304 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:34.788222    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs podName:15a5076c-b542-4a0d-bad7-cc06b81bbe3e nodeName:}" failed. No retries permitted until 2026-09-15 12:25:38.788199985 +0000 UTC m=+10.172910864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs") pod "network-metrics-daemon-bsb55" (UID: "15a5076c-b542-4a0d-bad7-cc06b81bbe3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:25:35.102315 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:35.101436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:35.102315 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:35.101574    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:25:35.102315 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:35.101643    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:35.102315 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:35.101732    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:25:36.099650 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:36.099214    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:36.099650 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:36.099347    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:37.103541 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:37.103506    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:37.104052 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:37.103652    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:25:37.104052 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:37.103693    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:37.104052 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:37.103765    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:25:38.100120 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:38.100033    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:38.100303 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:38.100168    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:38.720535 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:38.720484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5w4p\" (UniqueName: \"kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p\") pod \"network-check-target-n4m28\" (UID: \"2b715742-769a-4263-a81d-afd9e2c29183\") " pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:38.721058 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:38.720555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:38.721058 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:38.720734    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:25:38.721058 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:38.720801    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret podName:f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:46.720780639 +0000 UTC m=+18.105491511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret") pod "global-pull-secret-syncer-6txlk" (UID: "f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:25:38.721058 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:38.720892    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:25:38.721058 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:38.720926    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:25:38.721058 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:38.720940    2579 projected.go:196] Error preparing data for projected volume kube-api-access-q5w4p for pod openshift-network-diagnostics/network-check-target-n4m28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:25:38.721058 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:38.720978    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p podName:2b715742-769a-4263-a81d-afd9e2c29183 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:46.720966417 +0000 UTC m=+18.105677285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q5w4p" (UniqueName: "kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p") pod "network-check-target-n4m28" (UID: "2b715742-769a-4263-a81d-afd9e2c29183") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:25:38.821102 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:38.820965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs\") pod \"network-metrics-daemon-bsb55\" (UID: \"15a5076c-b542-4a0d-bad7-cc06b81bbe3e\") " pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:38.821293 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:38.821117    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:25:38.821293 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:38.821200    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs podName:15a5076c-b542-4a0d-bad7-cc06b81bbe3e nodeName:}" failed. No retries permitted until 2026-09-15 12:25:46.821178031 +0000 UTC m=+18.205888899 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs") pod "network-metrics-daemon-bsb55" (UID: "15a5076c-b542-4a0d-bad7-cc06b81bbe3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:25:39.100697 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:39.100135    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:39.100697 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:39.100243    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:25:39.100697 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:39.100567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:39.100697 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:39.100672    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:25:40.099328 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:40.099292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:40.099873 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:40.099444    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:41.099636 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:41.099503    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:41.100142 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:41.099651    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:25:41.100142 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:41.099807    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:41.100142 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:41.099966    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:25:42.099637 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:42.099601    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:42.100163 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:42.099736    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:43.099389 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:43.099348    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:43.099568 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:43.099349    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:43.099568 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:43.099495    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:25:43.099568 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:43.099521    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:25:44.099353 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:44.099320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:44.099536 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:44.099444    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:45.103162 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:45.103133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:45.103609 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:45.103247    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:25:45.103609 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:45.103299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:45.103609 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:45.103384    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:25:46.099447 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:46.099410    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:46.099622 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:46.099554    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:46.780080 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:46.780041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:46.780553 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:46.780133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5w4p\" (UniqueName: \"kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p\") pod \"network-check-target-n4m28\" (UID: \"2b715742-769a-4263-a81d-afd9e2c29183\") " pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:46.780553 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:46.780194    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:25:46.780553 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:46.780231    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:25:46.780553 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:46.780249    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:25:46.780553 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:46.780258    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret podName:f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:02.780239079 +0000 UTC m=+34.164949954 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret") pod "global-pull-secret-syncer-6txlk" (UID: "f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:25:46.780553 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:46.780260    2579 projected.go:196] Error preparing data for projected volume kube-api-access-q5w4p for pod openshift-network-diagnostics/network-check-target-n4m28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:25:46.780553 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:46.780297    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p podName:2b715742-769a-4263-a81d-afd9e2c29183 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:02.780285471 +0000 UTC m=+34.164996334 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q5w4p" (UniqueName: "kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p") pod "network-check-target-n4m28" (UID: "2b715742-769a-4263-a81d-afd9e2c29183") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:25:46.881186 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:46.881154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs\") pod \"network-metrics-daemon-bsb55\" (UID: \"15a5076c-b542-4a0d-bad7-cc06b81bbe3e\") " pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:46.881352 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:46.881292    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:25:46.881352 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:46.881348    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs podName:15a5076c-b542-4a0d-bad7-cc06b81bbe3e nodeName:}" failed. No retries permitted until 2026-09-15 12:26:02.88133076 +0000 UTC m=+34.266041631 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs") pod "network-metrics-daemon-bsb55" (UID: "15a5076c-b542-4a0d-bad7-cc06b81bbe3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:25:47.101519 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:47.101453    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:47.101660 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:47.101588    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:25:47.101660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:47.101635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:47.101811 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:47.101732    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:25:48.099386 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:48.099352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:48.099967 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:48.099451    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:49.099605 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:49.099573    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:49.100094 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:49.099653    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:49.100094 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:49.099671    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:25:49.100094 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:49.099773    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:25:50.099316 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:50.099276    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:50.099498 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:50.099403    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:51.099364 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:51.099345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:51.099765 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:51.099345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:51.099765 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:51.099474    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:25:51.099765 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:51.099518    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:25:51.186956 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:51.186932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr" event={"ID":"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139","Type":"ContainerStarted","Data":"81f24c6bcffe574c73a07945380403d1cba0a31ebb20d6e3657a415d780ecfa6"}
Sep 15 12:25:52.099726 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.099402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:52.100393 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:52.099807    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:52.193686 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.193655    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:25:52.194070 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.194043    2579 generic.go:356] "Generic (PLEG): container finished" podID="371b2ab6-5733-4e72-80bf-36ddbdc2ea74" containerID="e9b282832e8d09ff4b37157bcde949b411092b5864624210d35a9fcff0ecf822" exitCode=1
Sep 15 12:25:52.194158 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.194122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg" event={"ID":"371b2ab6-5733-4e72-80bf-36ddbdc2ea74","Type":"ContainerStarted","Data":"93dee42e76106441a4e7341e9cafae1c1cdaa7b19e0e95c93520d636bee99737"}
Sep 15 12:25:52.194217 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.194200    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg" event={"ID":"371b2ab6-5733-4e72-80bf-36ddbdc2ea74","Type":"ContainerStarted","Data":"f4327f207eeb7621d648bf692bca738ebef0ff139dab924e1cf074b18830508a"}
Sep 15 12:25:52.194268 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.194240    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg" event={"ID":"371b2ab6-5733-4e72-80bf-36ddbdc2ea74","Type":"ContainerStarted","Data":"ba9f0bb2c008bc400ce1dcbdd164e4790a6d19df8b3fcd4852102e853c6b93ce"}
Sep 15 12:25:52.194316 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.194282    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg" event={"ID":"371b2ab6-5733-4e72-80bf-36ddbdc2ea74","Type":"ContainerStarted","Data":"0c85bf36c24b783cf872cdbbb4d3a01948bf5ae28cb7def398892cd8e27a9237"}
Sep 15 12:25:52.194365 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.194330    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg" event={"ID":"371b2ab6-5733-4e72-80bf-36ddbdc2ea74","Type":"ContainerDied","Data":"e9b282832e8d09ff4b37157bcde949b411092b5864624210d35a9fcff0ecf822"}
Sep 15 12:25:52.194417 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.194381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg" event={"ID":"371b2ab6-5733-4e72-80bf-36ddbdc2ea74","Type":"ContainerStarted","Data":"d0c77e9173d204fd1c5af5e221dd50fdb5e2245b8e4074dcf6a26927a3f21c0b"}
Sep 15 12:25:52.195760 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.195734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xbbzp" event={"ID":"6fe7f374-634a-4074-97b1-21a380d41caa","Type":"ContainerStarted","Data":"cafd7155f1efa4a39afb66fc41d4d5b629486fc9e3c2b679174b70d1a60627f1"}
Sep 15 12:25:52.197348 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.197317    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4pc2d" event={"ID":"e17c9098-956f-4d77-a535-5b09749f3a95","Type":"ContainerStarted","Data":"95a1af1a264b7709dd6d7725ab2d8426bfa899f3fcfcefa55f099518561b0c43"}
Sep 15 12:25:52.199003 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.198971    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rdsrq" event={"ID":"594efe57-d61c-4858-8df4-41ebf3957073","Type":"ContainerStarted","Data":"985fa5b8d90b79c49e2ff4fe7ba460e185cfaeb0ed1858fdc18966a7d705ed05"}
Sep 15 12:25:52.200660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.200639    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-66lhz" event={"ID":"866085d7-f0ed-4104-bd66-d5964b6619c9","Type":"ContainerStarted","Data":"75cee34b98f419b61554afd4b6c85af5d8c57f957c57554fdebb97b53fae05dd"}
Sep 15 12:25:52.202192 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.202170    2579 generic.go:356] "Generic (PLEG): container finished" podID="ba6038d6a5b6f0c809b6edabd63fb379" containerID="1e5c8785d8b6a8490c0481b191ee517fd04bec97faff343aef708eeb009d339b" exitCode=0
Sep 15 12:25:52.202275 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.202250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal" event={"ID":"ba6038d6a5b6f0c809b6edabd63fb379","Type":"ContainerDied","Data":"1e5c8785d8b6a8490c0481b191ee517fd04bec97faff343aef708eeb009d339b"}
Sep 15 12:25:52.203879 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.203857    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-blcfn" event={"ID":"1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc","Type":"ContainerStarted","Data":"bad5c76bafa3c07338f6e545b4c2967c69d7b4b599a8cdce71124d8f81d0159e"}
Sep 15 12:25:52.207722 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.207704    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c05cc62-0238-4991-96af-7633ec6f0e05" containerID="c19ef8d3bb298fabf5f231fb5e4d93951756846783e241ae3e6adeab498f0d41" exitCode=0
Sep 15 12:25:52.207799 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.207774    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ht8cc" event={"ID":"6c05cc62-0238-4991-96af-7633ec6f0e05","Type":"ContainerDied","Data":"c19ef8d3bb298fabf5f231fb5e4d93951756846783e241ae3e6adeab498f0d41"}
Sep 15 12:25:52.209424 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.209403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-17.ec2.internal" event={"ID":"0eab95c6c565e847818c125e64a2e65f","Type":"ContainerStarted","Data":"7407e9a995b94e36865d8f3f401fdf85ea3dbf2a9b4e5b2ddbcc059da8acd800"}
Sep 15 12:25:52.212349 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.212217    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xbbzp" podStartSLOduration=2.964109901 podStartE2EDuration="22.212201659s" totalImagesPullingTime="19.248091758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:52.211461579 +0000 UTC m=+23.596172466" watchObservedRunningTime="2026-09-15 12:25:52.212201659 +0000 UTC m=+23.596912564"
Sep 15 12:25:52.238511 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.238462    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-66lhz" podStartSLOduration=3.390228584 podStartE2EDuration="22.238445728s" totalImagesPullingTime="18.848217144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:52.238313677 +0000 UTC m=+23.623024563" watchObservedRunningTime="2026-09-15 12:25:52.238445728 +0000 UTC m=+23.623156613"
Sep 15 12:25:52.252791 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.252735    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-17.ec2.internal" podStartSLOduration=21.252718317 podStartE2EDuration="21.252718317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:52.251626433 +0000 UTC m=+23.636337374" watchObservedRunningTime="2026-09-15 12:25:52.252718317 +0000 UTC m=+23.637429203"
Sep 15 12:25:52.287460 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.287422    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rdsrq" podStartSLOduration=3.268927414 podStartE2EDuration="22.287403805s" totalImagesPullingTime="19.018476391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:52.269329225 +0000 UTC m=+23.654040120" watchObservedRunningTime="2026-09-15 12:25:52.287403805 +0000 UTC m=+23.672114692"
Sep 15 12:25:52.304650 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.304609    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4pc2d" podStartSLOduration=10.941667632 podStartE2EDuration="22.304593474s" totalImagesPullingTime="11.362925842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:52.287366812 +0000 UTC m=+23.672077697" watchObservedRunningTime="2026-09-15 12:25:52.304593474 +0000 UTC m=+23.689304357"
Sep 15 12:25:52.304852 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.304813    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-blcfn" podStartSLOduration=3.381549537 podStartE2EDuration="22.304808677s" totalImagesPullingTime="18.92325914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:52.304385889 +0000 UTC m=+23.689096774" watchObservedRunningTime="2026-09-15 12:25:52.304808677 +0000 UTC m=+23.689519561"
Sep 15 12:25:52.392329 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.392295    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4pc2d"
Sep 15 12:25:52.393005 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:52.392982    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4pc2d"
Sep 15 12:25:53.102188 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:53.102166    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:53.102188 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:53.102177    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:53.102986 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:53.102280    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:25:53.102986 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:53.102407    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:25:53.213318 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:53.213252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr" event={"ID":"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139","Type":"ContainerStarted","Data":"aab76be368f6e4c5c4c2e6b75ee8c87a628b0522418d7758ac4c62e40ecc4a21"}
Sep 15 12:25:53.214998 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:53.214974    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gvtvg" event={"ID":"0c0fc181-a9d3-4e90-ae03-90b1d11d4146","Type":"ContainerStarted","Data":"952cde5a4ad59ad8f91686248063973a6983c10139b5fba0f0b18f3e142bfe39"}
Sep 15 12:25:53.217338 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:53.217276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal" event={"ID":"ba6038d6a5b6f0c809b6edabd63fb379","Type":"ContainerStarted","Data":"358ebfed4a770cc872bcdaecd4325471bfd8f061823c696f2e99c7f7119f4993"}
Sep 15 12:25:53.233496 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:53.233451    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gvtvg" podStartSLOduration=4.265037275 podStartE2EDuration="23.233437948s" totalImagesPullingTime="18.968400673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:53.233398406 +0000 UTC m=+24.618109292" watchObservedRunningTime="2026-09-15 12:25:53.233437948 +0000 UTC m=+24.618148830"
Sep 15 12:25:53.249367 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:53.249327    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-17.ec2.internal" podStartSLOduration=22.249313806 podStartE2EDuration="22.249313806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:53.248883737 +0000 UTC m=+24.633594624" watchObservedRunningTime="2026-09-15 12:25:53.249313806 +0000 UTC m=+24.634024690"
Sep 15 12:25:53.492350 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:53.492267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:53.582612 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:53.582580    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:53.647189 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:53.647155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:54.099178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:54.099103    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:54.099360 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:54.099229    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:54.223418 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:54.223396    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:25:54.223898 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:54.223878    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 12:25:54.223982 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:54.223893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg" event={"ID":"371b2ab6-5733-4e72-80bf-36ddbdc2ea74","Type":"ContainerStarted","Data":"f05aa6c6ce7996d653d0d30037ee1bb068e1532daac9ed230500016964c4a48a"}
Sep 15 12:25:54.254343 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:54.254124    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 12:25:54.918968 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:54.918931    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:55.103207 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:55.103178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:55.103207 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:55.103204    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:55.103476 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:55.103324    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:25:55.103476 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:55.103423    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:25:55.110768 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:55.110683    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T12:25:54.254341778Z","UUID":"41150b2b-a8b9-4ac6-ad12-2312a12395de","Handler":null,"Name":"","Endpoint":""}
Sep 15 12:25:55.112445 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:55.112430    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 12:25:55.112572 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:55.112459    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 12:25:55.174386 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:55.174323    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:55.228047 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:55.228014    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr" event={"ID":"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139","Type":"ContainerStarted","Data":"96cedb33c2ff7522c38cd4673f067131cd71d30356f07d50c44838a815933f86"}
Sep 15 12:25:55.228442 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:55.228351    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:55.972345 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:55.972310    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:56.027203 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:56.027168    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:56.099023 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:56.098988    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:56.099196 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:56.099101    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:57.042115 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:57.042091    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:57.099389 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:57.099278    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:57.099389 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:57.099305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:57.099671 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:57.099385    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:25:57.099671 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:57.099541    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:25:57.142884 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:57.142858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:57.240037 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:57.239971    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:25:57.240331 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:57.240307    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg" event={"ID":"371b2ab6-5733-4e72-80bf-36ddbdc2ea74","Type":"ContainerStarted","Data":"52dae11747216e1de6fa055b0f588e79224b5ef5643788977c4e2ebc5bb29c5e"}
Sep 15 12:25:57.240718 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:57.240686    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:57.240884 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:57.240870    2579 scope.go:117] "RemoveContainer" containerID="e9b282832e8d09ff4b37157bcde949b411092b5864624210d35a9fcff0ecf822"
Sep 15 12:25:57.255519 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:57.255494    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:58.049314 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.049069    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4pc2d"
Sep 15 12:25:58.049757 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.049404    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 12:25:58.049757 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.049568    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4pc2d"
Sep 15 12:25:58.099392 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.099363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:58.099511 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:58.099458    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:58.113498 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.113466    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:58.246915 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.246824    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:25:58.247340 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.247314    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg" event={"ID":"371b2ab6-5733-4e72-80bf-36ddbdc2ea74","Type":"ContainerStarted","Data":"88e8006b473fb3f74327aca07d09ad05c8fd028ec2394f1975c16f2c1cd9fa54"}
Sep 15 12:25:58.247573 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.247551    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:58.247722 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.247628    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:58.250035 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.250008    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c05cc62-0238-4991-96af-7633ec6f0e05" containerID="0e157a021564247232c216062fa6a3dfca82fbdddf9d8ed6929e61d222a1de78" exitCode=0
Sep 15 12:25:58.250218 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.250196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ht8cc" event={"ID":"6c05cc62-0238-4991-96af-7633ec6f0e05","Type":"ContainerDied","Data":"0e157a021564247232c216062fa6a3dfca82fbdddf9d8ed6929e61d222a1de78"}
Sep 15 12:25:58.268034 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.268013    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:25:58.275132 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.275075    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg" podStartSLOduration=9.32612827 podStartE2EDuration="28.275052383s" totalImagesPullingTime="18.948924113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:58.273534085 +0000 UTC m=+29.658244972" watchObservedRunningTime="2026-09-15 12:25:58.275052383 +0000 UTC m=+29.659763268"
Sep 15 12:25:58.520989 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.520821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:58.578135 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.578102    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:58.872752 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.872674    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:58.938677 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.938642    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:58.970734 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.970696    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n4m28"]
Sep 15 12:25:58.970899 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.970825    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:25:58.970976 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:58.970936    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:25:58.974105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.974079    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6txlk"]
Sep 15 12:25:58.974234 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.974179    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:25:58.974298 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:58.974278    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:25:58.974699 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.974674    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bsb55"]
Sep 15 12:25:58.974826 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:58.974769    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:25:58.974914 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:25:58.974867    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:25:59.059371 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:59.059334    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:59.254367 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:59.254142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr" event={"ID":"a1d8ffc1-2c2e-4f1e-87b9-c88b4b0d5139","Type":"ContainerStarted","Data":"019090cd159b696f0eae7655110c3f2870da7e4a83d273c09555393ce1d99472"}
Sep 15 12:25:59.271305 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:59.271264    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcrcr" podStartSLOduration=21.861850916 podStartE2EDuration="29.271249868s" totalImagesPullingTime="7.409398952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:59.270752417 +0000 UTC m=+30.655463294" watchObservedRunningTime="2026-09-15 12:25:59.271249868 +0000 UTC m=+30.655960752"
Sep 15 12:25:59.573138 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:59.571889    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:59.670244 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:59.670220    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:59.771188 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:59.771154    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:59.871502 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:59.871412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:25:59.927386 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:25:59.927354    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:00.258878 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:00.258785    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c05cc62-0238-4991-96af-7633ec6f0e05" containerID="5124e052c7799e40f53d0885b1d972f78a351bc77e233833c7048e0dfe1a55cd" exitCode=0
Sep 15 12:26:00.258878 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:00.258864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ht8cc" event={"ID":"6c05cc62-0238-4991-96af-7633ec6f0e05","Type":"ContainerDied","Data":"5124e052c7799e40f53d0885b1d972f78a351bc77e233833c7048e0dfe1a55cd"}
Sep 15 12:26:01.100026 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:01.099998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:26:01.100199 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:01.100004    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:26:01.100199 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:01.100087    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:26:01.100199 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:01.100149    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:26:01.100199 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:01.100176    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:26:01.100397 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:01.100277    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:26:01.263701 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:01.263623    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c05cc62-0238-4991-96af-7633ec6f0e05" containerID="362e3eb0d05ae1b83e466244983b01de7afa947ff887e135fc71d6d109efe051" exitCode=0
Sep 15 12:26:01.263701 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:01.263664    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ht8cc" event={"ID":"6c05cc62-0238-4991-96af-7633ec6f0e05","Type":"ContainerDied","Data":"362e3eb0d05ae1b83e466244983b01de7afa947ff887e135fc71d6d109efe051"}
Sep 15 12:26:02.792760 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:02.792681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5w4p\" (UniqueName: \"kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p\") pod \"network-check-target-n4m28\" (UID: \"2b715742-769a-4263-a81d-afd9e2c29183\") " pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:26:02.792760 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:02.792722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:26:02.793152 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:02.792826    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 12:26:02.793152 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:02.792860    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 12:26:02.793152 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:02.792881    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 12:26:02.793152 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:02.792891    2579 projected.go:196] Error preparing data for projected volume kube-api-access-q5w4p for pod openshift-network-diagnostics/network-check-target-n4m28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:26:02.793152 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:02.792894    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret podName:f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:34.792881344 +0000 UTC m=+66.177592205 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret") pod "global-pull-secret-syncer-6txlk" (UID: "f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1") : object "kube-system"/"original-pull-secret" not registered
Sep 15 12:26:02.793152 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:02.792929    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p podName:2b715742-769a-4263-a81d-afd9e2c29183 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:34.792917257 +0000 UTC m=+66.177628123 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-q5w4p" (UniqueName: "kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p") pod "network-check-target-n4m28" (UID: "2b715742-769a-4263-a81d-afd9e2c29183") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 12:26:02.893520 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:02.893477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs\") pod \"network-metrics-daemon-bsb55\" (UID: \"15a5076c-b542-4a0d-bad7-cc06b81bbe3e\") " pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:26:02.893709 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:02.893653    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:26:02.893774 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:02.893728    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs podName:15a5076c-b542-4a0d-bad7-cc06b81bbe3e nodeName:}" failed. No retries permitted until 2026-09-15 12:26:34.89370865 +0000 UTC m=+66.278419569 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs") pod "network-metrics-daemon-bsb55" (UID: "15a5076c-b542-4a0d-bad7-cc06b81bbe3e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 12:26:03.099084 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.099050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:26:03.099249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.099050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:26:03.099249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.099188    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:26:03.099249 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:03.099181    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6txlk" podUID="f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1"
Sep 15 12:26:03.099420 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:03.099272    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bsb55" podUID="15a5076c-b542-4a0d-bad7-cc06b81bbe3e"
Sep 15 12:26:03.099420 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:03.099357    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4m28" podUID="2b715742-769a-4263-a81d-afd9e2c29183"
Sep 15 12:26:03.562963 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.562878    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:03.615902 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.615871    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:03.941070 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.941039    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-17.ec2.internal" event="NodeReady"
Sep 15 12:26:03.941506 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.941201    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 12:26:03.974795 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.974764    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57995fd64d-tj4cl"]
Sep 15 12:26:03.976958 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.976933    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:03.979545 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.979281    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 12:26:03.979545 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.979301    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 12:26:03.979545 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.979352    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 12:26:03.979545 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.979427    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4xr84\""
Sep 15 12:26:03.984782 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.984758    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 12:26:03.985660 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.985078    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bx6hs"]
Sep 15 12:26:03.986999 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.986978    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2ndsn"]
Sep 15 12:26:03.987120 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.987102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bx6hs"
Sep 15 12:26:03.988907 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.988890    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:03.989678 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.989653    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 12:26:03.989776 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.989680    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 12:26:03.990032 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.990013    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v4v6p\""
Sep 15 12:26:03.990145 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.990058    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 12:26:03.991118 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.991097    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 12:26:03.991547 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.991529    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 12:26:03.991641 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.991602    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57995fd64d-tj4cl"]
Sep 15 12:26:03.991901 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:03.991881    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pzg8w\""
Sep 15 12:26:04.002282 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.002233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bx6hs"]
Sep 15 12:26:04.003085 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.003064    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2ndsn"]
Sep 15 12:26:04.068392 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.068355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:04.102523 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.102487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7812e2fd-cb28-4988-a3e0-5d9433a853f1-config-volume\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:04.102691 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.102528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwpdl\" (UniqueName: \"kubernetes.io/projected/7812e2fd-cb28-4988-a3e0-5d9433a853f1-kube-api-access-xwpdl\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:04.102691 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.102563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bce2551d-652b-4087-a003-003897e1c854-ca-trust-extracted\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.102691 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.102617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bce2551d-652b-4087-a003-003897e1c854-installation-pull-secrets\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.102691 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.102682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40dd46b2-753c-40ea-a811-15944e4951b0-cert\") pod \"ingress-canary-bx6hs\" (UID: \"40dd46b2-753c-40ea-a811-15944e4951b0\") " pod="openshift-ingress-canary/ingress-canary-bx6hs"
Sep 15 12:26:04.102930 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.102790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bce2551d-652b-4087-a003-003897e1c854-image-registry-private-configuration\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.102930 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.102845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fdf9\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-kube-api-access-4fdf9\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.102930 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.102873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7812e2fd-cb28-4988-a3e0-5d9433a853f1-tmp-dir\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:04.102930 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.102911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bce2551d-652b-4087-a003-003897e1c854-trusted-ca\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.103108 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.102935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl59s\" (UniqueName: \"kubernetes.io/projected/40dd46b2-753c-40ea-a811-15944e4951b0-kube-api-access-dl59s\") pod \"ingress-canary-bx6hs\" (UID: \"40dd46b2-753c-40ea-a811-15944e4951b0\") " pod="openshift-ingress-canary/ingress-canary-bx6hs"
Sep 15 12:26:04.103108 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.102958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:04.103108 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.102999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-bound-sa-token\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.103108 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.103044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bce2551d-652b-4087-a003-003897e1c854-registry-certificates\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.103108 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.103075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.124281 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.124251    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:04.203453 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.203368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bce2551d-652b-4087-a003-003897e1c854-image-registry-private-configuration\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.203453 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.203413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fdf9\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-kube-api-access-4fdf9\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.203453 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.203436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7812e2fd-cb28-4988-a3e0-5d9433a853f1-tmp-dir\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:04.203715 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.203689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bce2551d-652b-4087-a003-003897e1c854-trusted-ca\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.203788 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.203770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7812e2fd-cb28-4988-a3e0-5d9433a853f1-tmp-dir\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:04.203866 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.203812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl59s\" (UniqueName: \"kubernetes.io/projected/40dd46b2-753c-40ea-a811-15944e4951b0-kube-api-access-dl59s\") pod \"ingress-canary-bx6hs\" (UID: \"40dd46b2-753c-40ea-a811-15944e4951b0\") " pod="openshift-ingress-canary/ingress-canary-bx6hs"
Sep 15 12:26:04.203866 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.203858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:04.204067 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.203902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-bound-sa-token\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.204120 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.204091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bce2551d-652b-4087-a003-003897e1c854-registry-certificates\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.204161 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.204120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.204161 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:04.204152    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 12:26:04.204315 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.204176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7812e2fd-cb28-4988-a3e0-5d9433a853f1-config-volume\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:04.205367 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:04.205339    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 12:26:04.205476 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:04.205361    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57995fd64d-tj4cl: secret "image-registry-tls" not found
Sep 15 12:26:04.205591 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.205569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7812e2fd-cb28-4988-a3e0-5d9433a853f1-config-volume\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:04.205854 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.204199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwpdl\" (UniqueName: \"kubernetes.io/projected/7812e2fd-cb28-4988-a3e0-5d9433a853f1-kube-api-access-xwpdl\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:04.205993 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:04.205976    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls podName:7812e2fd-cb28-4988-a3e0-5d9433a853f1 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:04.704195084 +0000 UTC m=+36.088905962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls") pod "dns-default-2ndsn" (UID: "7812e2fd-cb28-4988-a3e0-5d9433a853f1") : secret "dns-default-metrics-tls" not found
Sep 15 12:26:04.206071 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:04.206018    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls podName:bce2551d-652b-4087-a003-003897e1c854 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:04.705998606 +0000 UTC m=+36.090709484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls") pod "image-registry-57995fd64d-tj4cl" (UID: "bce2551d-652b-4087-a003-003897e1c854") : secret "image-registry-tls" not found
Sep 15 12:26:04.206071 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.206063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bce2551d-652b-4087-a003-003897e1c854-ca-trust-extracted\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.206169 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.206097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bce2551d-652b-4087-a003-003897e1c854-installation-pull-secrets\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.206271 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.206252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40dd46b2-753c-40ea-a811-15944e4951b0-cert\") pod \"ingress-canary-bx6hs\" (UID: \"40dd46b2-753c-40ea-a811-15944e4951b0\") " pod="openshift-ingress-canary/ingress-canary-bx6hs"
Sep 15 12:26:04.206739 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.206715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bce2551d-652b-4087-a003-003897e1c854-registry-certificates\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.206849 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.206207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bce2551d-652b-4087-a003-003897e1c854-trusted-ca\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.207076 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.207052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bce2551d-652b-4087-a003-003897e1c854-ca-trust-extracted\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.209288 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.208888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bce2551d-652b-4087-a003-003897e1c854-image-registry-private-configuration\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.210103 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.210080    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bce2551d-652b-4087-a003-003897e1c854-installation-pull-secrets\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.211604 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.211580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40dd46b2-753c-40ea-a811-15944e4951b0-cert\") pod \"ingress-canary-bx6hs\" (UID: \"40dd46b2-753c-40ea-a811-15944e4951b0\") " pod="openshift-ingress-canary/ingress-canary-bx6hs"
Sep 15 12:26:04.213439 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.213399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwpdl\" (UniqueName: \"kubernetes.io/projected/7812e2fd-cb28-4988-a3e0-5d9433a853f1-kube-api-access-xwpdl\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:04.213552 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.213530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fdf9\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-kube-api-access-4fdf9\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.214193 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.214171    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl59s\" (UniqueName: \"kubernetes.io/projected/40dd46b2-753c-40ea-a811-15944e4951b0-kube-api-access-dl59s\") pod \"ingress-canary-bx6hs\" (UID: \"40dd46b2-753c-40ea-a811-15944e4951b0\") " pod="openshift-ingress-canary/ingress-canary-bx6hs"
Sep 15 12:26:04.216027 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.216008    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-bound-sa-token\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.300095 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.300059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bx6hs"
Sep 15 12:26:04.332789 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.332579    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:04.425396 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.425353    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:04.494862 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.494769    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:04.561086 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.561042    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:04.580508 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.580481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bx6hs"]
Sep 15 12:26:04.710237 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.710199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:04.710433 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.710256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:04.710433 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:04.710374    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 12:26:04.710546 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:04.710456    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls podName:7812e2fd-cb28-4988-a3e0-5d9433a853f1 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:05.710429997 +0000 UTC m=+37.095140882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls") pod "dns-default-2ndsn" (UID: "7812e2fd-cb28-4988-a3e0-5d9433a853f1") : secret "dns-default-metrics-tls" not found
Sep 15 12:26:04.710546 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:04.710377    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 12:26:04.710546 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:04.710493    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57995fd64d-tj4cl: secret "image-registry-tls" not found
Sep 15 12:26:04.710700 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:04.710549    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls podName:bce2551d-652b-4087-a003-003897e1c854 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:05.710533568 +0000 UTC m=+37.095244431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls") pod "image-registry-57995fd64d-tj4cl" (UID: "bce2551d-652b-4087-a003-003897e1c854") : secret "image-registry-tls" not found
Sep 15 12:26:04.772147 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.772051    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:04.872896 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:04.872854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:05.100022 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.099989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:26:05.100463 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.099989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:26:05.100463 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.099989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:26:05.103667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.103520    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 12:26:05.103667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.103534    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 12:26:05.103667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.103547    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mnpg9\""
Sep 15 12:26:05.103667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.103564    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 12:26:05.103667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.103574    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j4plp\""
Sep 15 12:26:05.103667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.103625    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 12:26:05.327561 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.327526    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-fmtjg"]
Sep 15 12:26:05.333068 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.333046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fmtjg"
Sep 15 12:26:05.335298 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.335272    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 15 12:26:05.335452 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.335328    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-67bg6\""
Sep 15 12:26:05.335533 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.335334    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 15 12:26:05.337587 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.337562    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-fmtjg"]
Sep 15 12:26:05.416654 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.416575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz9x6\" (UniqueName: \"kubernetes.io/projected/4f01962c-d997-4add-9e2b-0a46caea55d0-kube-api-access-jz9x6\") pod \"migrator-8b4d697c5-fmtjg\" (UID: \"4f01962c-d997-4add-9e2b-0a46caea55d0\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fmtjg"
Sep 15 12:26:05.517313 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.517276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz9x6\" (UniqueName: \"kubernetes.io/projected/4f01962c-d997-4add-9e2b-0a46caea55d0-kube-api-access-jz9x6\") pod \"migrator-8b4d697c5-fmtjg\" (UID: \"4f01962c-d997-4add-9e2b-0a46caea55d0\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fmtjg"
Sep 15 12:26:05.526519 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.526483    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz9x6\" (UniqueName: \"kubernetes.io/projected/4f01962c-d997-4add-9e2b-0a46caea55d0-kube-api-access-jz9x6\") pod \"migrator-8b4d697c5-fmtjg\" (UID: \"4f01962c-d997-4add-9e2b-0a46caea55d0\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fmtjg"
Sep 15 12:26:05.644634 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.644597    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fmtjg"
Sep 15 12:26:05.719247 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.719151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:05.719413 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:05.719326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:05.719413 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:05.719332    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 12:26:05.719413 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:05.719356    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57995fd64d-tj4cl: secret "image-registry-tls" not found
Sep 15 12:26:05.719564 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:05.719419    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls podName:bce2551d-652b-4087-a003-003897e1c854 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:07.719395704 +0000 UTC m=+39.104106584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls") pod "image-registry-57995fd64d-tj4cl" (UID: "bce2551d-652b-4087-a003-003897e1c854") : secret "image-registry-tls" not found
Sep 15 12:26:05.719564 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:05.719420    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 12:26:05.719564 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:05.719460    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls podName:7812e2fd-cb28-4988-a3e0-5d9433a853f1 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:07.719449312 +0000 UTC m=+39.104160195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls") pod "dns-default-2ndsn" (UID: "7812e2fd-cb28-4988-a3e0-5d9433a853f1") : secret "dns-default-metrics-tls" not found
Sep 15 12:26:05.992786 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:26:05.992717    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40dd46b2_753c_40ea_a811_15944e4951b0.slice/crio-24a63682ca3bef43c2ce70a0c1a9e36a561304cb1d7e658fc2a2b0ea740787df WatchSource:0}: Error finding container 24a63682ca3bef43c2ce70a0c1a9e36a561304cb1d7e658fc2a2b0ea740787df: Status 404 returned error can't find the container with id 24a63682ca3bef43c2ce70a0c1a9e36a561304cb1d7e658fc2a2b0ea740787df
Sep 15 12:26:06.256687 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.256604    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-fmtjg"]
Sep 15 12:26:06.276123 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.276090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bx6hs" event={"ID":"40dd46b2-753c-40ea-a811-15944e4951b0","Type":"ContainerStarted","Data":"24a63682ca3bef43c2ce70a0c1a9e36a561304cb1d7e658fc2a2b0ea740787df"}
Sep 15 12:26:06.323051 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:26:06.323023    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f01962c_d997_4add_9e2b_0a46caea55d0.slice/crio-2459b58cec6f405910b354431087f55a2be0971e578c6a9a37bc81bfca6de434 WatchSource:0}: Error finding container 2459b58cec6f405910b354431087f55a2be0971e578c6a9a37bc81bfca6de434: Status 404 returned error can't find the container with id 2459b58cec6f405910b354431087f55a2be0971e578c6a9a37bc81bfca6de434
Sep 15 12:26:06.360752 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.360730    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-f5dzm"]
Sep 15 12:26:06.384229 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.384203    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-f5dzm"]
Sep 15 12:26:06.384329 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.384266    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-f5dzm"
Sep 15 12:26:06.386802 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.386777    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 15 12:26:06.386976 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.386924    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9ftdl\""
Sep 15 12:26:06.387089 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.387071    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 15 12:26:06.388011 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.387814    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 15 12:26:06.388011 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.387915    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 15 12:26:06.526384 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.526356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/91d94219-1cf9-427a-8425-d4b9a4c2b4f3-signing-cabundle\") pod \"service-ca-568869fdc5-f5dzm\" (UID: \"91d94219-1cf9-427a-8425-d4b9a4c2b4f3\") " pod="openshift-service-ca/service-ca-568869fdc5-f5dzm"
Sep 15 12:26:06.526525 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.526410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/91d94219-1cf9-427a-8425-d4b9a4c2b4f3-signing-key\") pod \"service-ca-568869fdc5-f5dzm\" (UID: \"91d94219-1cf9-427a-8425-d4b9a4c2b4f3\") " pod="openshift-service-ca/service-ca-568869fdc5-f5dzm"
Sep 15 12:26:06.526525 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.526500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9cbl\" (UniqueName: \"kubernetes.io/projected/91d94219-1cf9-427a-8425-d4b9a4c2b4f3-kube-api-access-p9cbl\") pod \"service-ca-568869fdc5-f5dzm\" (UID: \"91d94219-1cf9-427a-8425-d4b9a4c2b4f3\") " pod="openshift-service-ca/service-ca-568869fdc5-f5dzm"
Sep 15 12:26:06.627315 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.627288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9cbl\" (UniqueName: \"kubernetes.io/projected/91d94219-1cf9-427a-8425-d4b9a4c2b4f3-kube-api-access-p9cbl\") pod \"service-ca-568869fdc5-f5dzm\" (UID: \"91d94219-1cf9-427a-8425-d4b9a4c2b4f3\") " pod="openshift-service-ca/service-ca-568869fdc5-f5dzm"
Sep 15 12:26:06.627430 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.627414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/91d94219-1cf9-427a-8425-d4b9a4c2b4f3-signing-cabundle\") pod \"service-ca-568869fdc5-f5dzm\" (UID: \"91d94219-1cf9-427a-8425-d4b9a4c2b4f3\") " pod="openshift-service-ca/service-ca-568869fdc5-f5dzm"
Sep 15 12:26:06.627474 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.627450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/91d94219-1cf9-427a-8425-d4b9a4c2b4f3-signing-key\") pod \"service-ca-568869fdc5-f5dzm\" (UID: \"91d94219-1cf9-427a-8425-d4b9a4c2b4f3\") " pod="openshift-service-ca/service-ca-568869fdc5-f5dzm"
Sep 15 12:26:06.628248 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.628226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/91d94219-1cf9-427a-8425-d4b9a4c2b4f3-signing-cabundle\") pod \"service-ca-568869fdc5-f5dzm\" (UID: \"91d94219-1cf9-427a-8425-d4b9a4c2b4f3\") " pod="openshift-service-ca/service-ca-568869fdc5-f5dzm"
Sep 15 12:26:06.629914 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.629891    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/91d94219-1cf9-427a-8425-d4b9a4c2b4f3-signing-key\") pod \"service-ca-568869fdc5-f5dzm\" (UID: \"91d94219-1cf9-427a-8425-d4b9a4c2b4f3\") " pod="openshift-service-ca/service-ca-568869fdc5-f5dzm"
Sep 15 12:26:06.635193 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.635166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9cbl\" (UniqueName: \"kubernetes.io/projected/91d94219-1cf9-427a-8425-d4b9a4c2b4f3-kube-api-access-p9cbl\") pod \"service-ca-568869fdc5-f5dzm\" (UID: \"91d94219-1cf9-427a-8425-d4b9a4c2b4f3\") " pod="openshift-service-ca/service-ca-568869fdc5-f5dzm"
Sep 15 12:26:06.695335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.695305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-f5dzm"
Sep 15 12:26:06.980681 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:06.980630    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-f5dzm"]
Sep 15 12:26:06.985156 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:26:06.985123    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d94219_1cf9_427a_8425_d4b9a4c2b4f3.slice/crio-1c07ecda238db931cc456aebb581e6820775db98789d3e00449be2446fc09323 WatchSource:0}: Error finding container 1c07ecda238db931cc456aebb581e6820775db98789d3e00449be2446fc09323: Status 404 returned error can't find the container with id 1c07ecda238db931cc456aebb581e6820775db98789d3e00449be2446fc09323
Sep 15 12:26:07.280309 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:07.280264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-f5dzm" event={"ID":"91d94219-1cf9-427a-8425-d4b9a4c2b4f3","Type":"ContainerStarted","Data":"1c07ecda238db931cc456aebb581e6820775db98789d3e00449be2446fc09323"}
Sep 15 12:26:07.281528 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:07.281497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fmtjg" event={"ID":"4f01962c-d997-4add-9e2b-0a46caea55d0","Type":"ContainerStarted","Data":"2459b58cec6f405910b354431087f55a2be0971e578c6a9a37bc81bfca6de434"}
Sep 15 12:26:07.285217 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:07.285185    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c05cc62-0238-4991-96af-7633ec6f0e05" containerID="c49b85ec31447405d0e46e98c36a21b5ddbc3846882e716e2333896fb3f8fd82" exitCode=0
Sep 15 12:26:07.285350 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:07.285231    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ht8cc" event={"ID":"6c05cc62-0238-4991-96af-7633ec6f0e05","Type":"ContainerDied","Data":"c49b85ec31447405d0e46e98c36a21b5ddbc3846882e716e2333896fb3f8fd82"}
Sep 15 12:26:07.737982 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:07.737933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:07.738161 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:07.738063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:07.738161 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:07.738150    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 12:26:07.738281 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:07.738172    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57995fd64d-tj4cl: secret "image-registry-tls" not found
Sep 15 12:26:07.738281 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:07.738191    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 12:26:07.738281 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:07.738266    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls podName:bce2551d-652b-4087-a003-003897e1c854 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:11.738218287 +0000 UTC m=+43.122929164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls") pod "image-registry-57995fd64d-tj4cl" (UID: "bce2551d-652b-4087-a003-003897e1c854") : secret "image-registry-tls" not found
Sep 15 12:26:07.738451 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:07.738287    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls podName:7812e2fd-cb28-4988-a3e0-5d9433a853f1 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:11.738275774 +0000 UTC m=+43.122986644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls") pod "dns-default-2ndsn" (UID: "7812e2fd-cb28-4988-a3e0-5d9433a853f1") : secret "dns-default-metrics-tls" not found
Sep 15 12:26:08.294036 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:08.293957    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c05cc62-0238-4991-96af-7633ec6f0e05" containerID="01b705565d56b860404a04f353039baaa0da5a4299d337ee01fa2d56b7d1f2fa" exitCode=0
Sep 15 12:26:08.294036 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:08.294021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ht8cc" event={"ID":"6c05cc62-0238-4991-96af-7633ec6f0e05","Type":"ContainerDied","Data":"01b705565d56b860404a04f353039baaa0da5a4299d337ee01fa2d56b7d1f2fa"}
Sep 15 12:26:08.724505 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:08.724463    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:08.782512 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:08.782475    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:09.584193 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:09.584161    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:09.643743 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:09.643713    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:10.157112 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:10.156970    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:10.301739 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:10.301707    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fmtjg" event={"ID":"4f01962c-d997-4add-9e2b-0a46caea55d0","Type":"ContainerStarted","Data":"0a6d87f39f21a0086ad4c1258fe4c7185d49d0914f79e9e73021e10b38e5dd45"}
Sep 15 12:26:10.301900 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:10.301804    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fmtjg" event={"ID":"4f01962c-d997-4add-9e2b-0a46caea55d0","Type":"ContainerStarted","Data":"2faf1b4cde9050a63e73db853514440fb80f9041f89895c707a1dae80e88c6a0"}
Sep 15 12:26:10.305333 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:10.305307    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ht8cc" event={"ID":"6c05cc62-0238-4991-96af-7633ec6f0e05","Type":"ContainerStarted","Data":"14ed8a848abba4df225fe9c7f4797f25155e4fcc9979b44470462850ff2dc06c"}
Sep 15 12:26:10.306875 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:10.306844    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bx6hs" event={"ID":"40dd46b2-753c-40ea-a811-15944e4951b0","Type":"ContainerStarted","Data":"ca4a648b2839f250641ed99e38e1a6148c7b9d0ff47607a6d9541ea39ce18270"}
Sep 15 12:26:10.308191 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:10.308167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-f5dzm" event={"ID":"91d94219-1cf9-427a-8425-d4b9a4c2b4f3","Type":"ContainerStarted","Data":"6a5f1759a77535f9791737436807ae1fe84d6cdbb3efa59b3448534bfefd0e62"}
Sep 15 12:26:10.318421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:10.318367    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-fmtjg" podStartSLOduration=1.679321613 podStartE2EDuration="5.318352416s" totalImagesPullingTime="3.639030803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:10.317195752 +0000 UTC m=+41.701906640" watchObservedRunningTime="2026-09-15 12:26:10.318352416 +0000 UTC m=+41.703063301"
Sep 15 12:26:10.356242 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:10.356199    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bx6hs" podStartSLOduration=3.3700530730000002 podStartE2EDuration="7.356184967s" totalImagesPullingTime="3.986131894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:10.333367282 +0000 UTC m=+41.718078167" watchObservedRunningTime="2026-09-15 12:26:10.356184967 +0000 UTC m=+41.740895851"
Sep 15 12:26:10.356516 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:10.356496    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ht8cc" podStartSLOduration=9.444571894 podStartE2EDuration="40.356490359s" totalImagesPullingTime="30.911918465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:10.354884355 +0000 UTC m=+41.739595253" watchObservedRunningTime="2026-09-15 12:26:10.356490359 +0000 UTC m=+41.741201242"
Sep 15 12:26:10.370304 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:10.370258    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-f5dzm" podStartSLOduration=1.375210979 podStartE2EDuration="4.370241071s" totalImagesPullingTime="2.995030092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:10.369999812 +0000 UTC m=+41.754710696" watchObservedRunningTime="2026-09-15 12:26:10.370241071 +0000 UTC m=+41.754951957"
Sep 15 12:26:11.542374 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:11.542338    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:11.775049 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:11.775010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:11.775222 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:11.775071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:11.775222 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:11.775178    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 12:26:11.775222 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:11.775207    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 12:26:11.775222 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:11.775221    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57995fd64d-tj4cl: secret "image-registry-tls" not found
Sep 15 12:26:11.775447 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:11.775261    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls podName:7812e2fd-cb28-4988-a3e0-5d9433a853f1 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:19.775239732 +0000 UTC m=+51.159950608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls") pod "dns-default-2ndsn" (UID: "7812e2fd-cb28-4988-a3e0-5d9433a853f1") : secret "dns-default-metrics-tls" not found
Sep 15 12:26:11.775447 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:11.775288    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls podName:bce2551d-652b-4087-a003-003897e1c854 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:19.775271448 +0000 UTC m=+51.159982323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls") pod "image-registry-57995fd64d-tj4cl" (UID: "bce2551d-652b-4087-a003-003897e1c854") : secret "image-registry-tls" not found
Sep 15 12:26:11.817182 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:11.817090    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:11.917429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:11.917396    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:12.017467 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:12.017430    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:12.126669 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:12.126634    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:12.220933 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:12.220885    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:12.320461 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:12.320423    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:12.843764 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:12.843719    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:13.246511 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:13.246477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:13.522661 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:13.522582    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:13.650623 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:13.650584    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:14.250694 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:14.250655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:14.563408 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:14.563315    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:14.664669 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:14.664628    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:14.729925 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:14.729896    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:14.860681 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:14.860648    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:15.037890 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:15.037852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:15.235463 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:15.235383    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:15.292992 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:15.292950    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:15.541974 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:15.541892    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:15.645486 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:15.645453    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:15.743244 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:15.743206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:16.242592 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:16.242555    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:16.344394 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:16.344305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:16.448687 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:16.448660    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:16.648906 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:16.648875    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:16.841548 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:16.841501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:17.065589 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:17.065507    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:17.125787 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:17.125754    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:17.243861 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:17.243807    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:17.441703 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:17.441668    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:17.871319 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:17.871273    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:18.442261 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:18.442230    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:18.883459 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:18.883418    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:19.137512 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:19.137425    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:19.247260 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:19.247219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:19.443526 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:19.443443    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:19.833429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:19.833333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:19.833606 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:19.833448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:19.836227 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:19.836201    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7812e2fd-cb28-4988-a3e0-5d9433a853f1-metrics-tls\") pod \"dns-default-2ndsn\" (UID: \"7812e2fd-cb28-4988-a3e0-5d9433a853f1\") " pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:19.836337 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:19.836283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls\") pod \"image-registry-57995fd64d-tj4cl\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") " pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:19.888195 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:19.888169    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:19.905321 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:19.905293    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:20.186984 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:20.186954    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57995fd64d-tj4cl"]
Sep 15 12:26:20.189896 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:26:20.189866    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce2551d_652b_4087_a003_003897e1c854.slice/crio-43476a985addc4100eb26c9c774546cb6d7e0bd800f1d9573b429f98a3faaa9d WatchSource:0}: Error finding container 43476a985addc4100eb26c9c774546cb6d7e0bd800f1d9573b429f98a3faaa9d: Status 404 returned error can't find the container with id 43476a985addc4100eb26c9c774546cb6d7e0bd800f1d9573b429f98a3faaa9d
Sep 15 12:26:20.216492 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:20.216470    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2ndsn"]
Sep 15 12:26:20.228847 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:26:20.228804    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7812e2fd_cb28_4988_a3e0_5d9433a853f1.slice/crio-80200b9bee31677657ec2a2ea604e8272f451231573374be97c5a93c49fee866 WatchSource:0}: Error finding container 80200b9bee31677657ec2a2ea604e8272f451231573374be97c5a93c49fee866: Status 404 returned error can't find the container with id 80200b9bee31677657ec2a2ea604e8272f451231573374be97c5a93c49fee866
Sep 15 12:26:20.336507 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:20.336472    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2ndsn" event={"ID":"7812e2fd-cb28-4988-a3e0-5d9433a853f1","Type":"ContainerStarted","Data":"80200b9bee31677657ec2a2ea604e8272f451231573374be97c5a93c49fee866"}
Sep 15 12:26:20.338082 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:20.338052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57995fd64d-tj4cl" event={"ID":"bce2551d-652b-4087-a003-003897e1c854","Type":"ContainerStarted","Data":"6313733ae2bcfc914ec9407925433afc113a33bbcb34fb0d0eb112577bd01b69"}
Sep 15 12:26:20.338197 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:20.338094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57995fd64d-tj4cl" event={"ID":"bce2551d-652b-4087-a003-003897e1c854","Type":"ContainerStarted","Data":"43476a985addc4100eb26c9c774546cb6d7e0bd800f1d9573b429f98a3faaa9d"}
Sep 15 12:26:20.338197 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:20.338188    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:20.357532 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:20.357484    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57995fd64d-tj4cl" podStartSLOduration=47.357469623 podStartE2EDuration="47.357469623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:20.356800673 +0000 UTC m=+51.741511558" watchObservedRunningTime="2026-09-15 12:26:20.357469623 +0000 UTC m=+51.742180507"
Sep 15 12:26:22.348008 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:22.347977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2ndsn" event={"ID":"7812e2fd-cb28-4988-a3e0-5d9433a853f1","Type":"ContainerStarted","Data":"39c4ea72891d92c8b9fba423dbf5c02855e4b7db09e8a9429d89df5b05e44036"}
Sep 15 12:26:22.348421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:22.348057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2ndsn" event={"ID":"7812e2fd-cb28-4988-a3e0-5d9433a853f1","Type":"ContainerStarted","Data":"38a8ae0773f37dbfdaac7f91d5cb25eb53961f99ee933176bf0fbbcf65f54f9b"}
Sep 15 12:26:22.348421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:22.348107    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:22.365803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:22.365761    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2ndsn" podStartSLOduration=17.629166694 podStartE2EDuration="19.365749655s" totalImagesPullingTime="1.736582961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:22.36457036 +0000 UTC m=+53.749281244" watchObservedRunningTime="2026-09-15 12:26:22.365749655 +0000 UTC m=+53.750460538"
Sep 15 12:26:23.476623 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:23.476583    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:23.575253 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:23.575224    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:24.267172 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:24.267137    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:24.329216 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:24.329177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:27.028518 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:27.028479    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:27.134076 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:27.134046    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:28.081364 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.081327    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-qccwt"]
Sep 15 12:26:28.111485 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.111452    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"]
Sep 15 12:26:28.111635 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.111561    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.115638 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.115612    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 12:26:28.115774 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.115687    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zkwxg\""
Sep 15 12:26:28.115774 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.115744    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 15 12:26:28.115774 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.115687    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 15 12:26:28.115959 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.115781    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 12:26:28.120427 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.120409    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 15 12:26:28.135313 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.135296    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-qccwt"]
Sep 15 12:26:28.135408 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.135324    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"]
Sep 15 12:26:28.135408 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.135339    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57995fd64d-tj4cl"]
Sep 15 12:26:28.135475 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.135411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"
Sep 15 12:26:28.138082 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.138062    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 15 12:26:28.138267 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.138251    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 15 12:26:28.138314 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.138295    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 15 12:26:28.138568 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.138552    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 15 12:26:28.183766 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.183739    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79b48c6d55-27vqd"]
Sep 15 12:26:28.191958 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.191937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a29db7b-7128-47ba-8169-b2dd5882a1f4-tmp\") pod \"klusterlet-addon-workmgr-cc5977694-7sbbc\" (UID: \"7a29db7b-7128-47ba-8169-b2dd5882a1f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"
Sep 15 12:26:28.192068 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.191964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49823d24-5865-4a85-bacb-195339de36b2-trusted-ca-bundle\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.192068 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.191982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49823d24-5865-4a85-bacb-195339de36b2-serving-cert\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.192068 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.192000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49823d24-5865-4a85-bacb-195339de36b2-tmp\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.192068 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.192046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gvl6\" (UniqueName: \"kubernetes.io/projected/49823d24-5865-4a85-bacb-195339de36b2-kube-api-access-5gvl6\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.192205 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.192083    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7a29db7b-7128-47ba-8169-b2dd5882a1f4-klusterlet-config\") pod \"klusterlet-addon-workmgr-cc5977694-7sbbc\" (UID: \"7a29db7b-7128-47ba-8169-b2dd5882a1f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"
Sep 15 12:26:28.192205 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.192143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fjs4\" (UniqueName: \"kubernetes.io/projected/7a29db7b-7128-47ba-8169-b2dd5882a1f4-kube-api-access-5fjs4\") pod \"klusterlet-addon-workmgr-cc5977694-7sbbc\" (UID: \"7a29db7b-7128-47ba-8169-b2dd5882a1f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"
Sep 15 12:26:28.192205 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.192172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/49823d24-5865-4a85-bacb-195339de36b2-snapshots\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.192302 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.192216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49823d24-5865-4a85-bacb-195339de36b2-service-ca-bundle\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.199968 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.199943    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:28.201264 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.201245    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.202864 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.202820    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79b48c6d55-27vqd"]
Sep 15 12:26:28.284191 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.284158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:28.293118 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49823d24-5865-4a85-bacb-195339de36b2-serving-cert\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.293261 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-image-registry-private-configuration\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.293261 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49823d24-5865-4a85-bacb-195339de36b2-tmp\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.293261 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-registry-tls\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.293261 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-registry-certificates\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.293261 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-installation-pull-secrets\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.293495 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fjs4\" (UniqueName: \"kubernetes.io/projected/7a29db7b-7128-47ba-8169-b2dd5882a1f4-kube-api-access-5fjs4\") pod \"klusterlet-addon-workmgr-cc5977694-7sbbc\" (UID: \"7a29db7b-7128-47ba-8169-b2dd5882a1f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"
Sep 15 12:26:28.293495 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/49823d24-5865-4a85-bacb-195339de36b2-snapshots\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.293495 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-trusted-ca\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.293643 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gvl6\" (UniqueName: \"kubernetes.io/projected/49823d24-5865-4a85-bacb-195339de36b2-kube-api-access-5gvl6\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.293643 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a29db7b-7128-47ba-8169-b2dd5882a1f4-tmp\") pod \"klusterlet-addon-workmgr-cc5977694-7sbbc\" (UID: \"7a29db7b-7128-47ba-8169-b2dd5882a1f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"
Sep 15 12:26:28.293643 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49823d24-5865-4a85-bacb-195339de36b2-trusted-ca-bundle\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.293643 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk684\" (UniqueName: \"kubernetes.io/projected/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-kube-api-access-bk684\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.293643 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293627    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49823d24-5865-4a85-bacb-195339de36b2-tmp\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.293916 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7a29db7b-7128-47ba-8169-b2dd5882a1f4-klusterlet-config\") pod \"klusterlet-addon-workmgr-cc5977694-7sbbc\" (UID: \"7a29db7b-7128-47ba-8169-b2dd5882a1f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"
Sep 15 12:26:28.293916 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-bound-sa-token\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.293916 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-ca-trust-extracted\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.293916 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49823d24-5865-4a85-bacb-195339de36b2-service-ca-bundle\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.293916 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.293906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a29db7b-7128-47ba-8169-b2dd5882a1f4-tmp\") pod \"klusterlet-addon-workmgr-cc5977694-7sbbc\" (UID: \"7a29db7b-7128-47ba-8169-b2dd5882a1f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"
Sep 15 12:26:28.294165 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.294004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/49823d24-5865-4a85-bacb-195339de36b2-snapshots\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.294336 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.294316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49823d24-5865-4a85-bacb-195339de36b2-trusted-ca-bundle\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.294420 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.294401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49823d24-5865-4a85-bacb-195339de36b2-service-ca-bundle\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.295723 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.295706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49823d24-5865-4a85-bacb-195339de36b2-serving-cert\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.296079 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.296062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7a29db7b-7128-47ba-8169-b2dd5882a1f4-klusterlet-config\") pod \"klusterlet-addon-workmgr-cc5977694-7sbbc\" (UID: \"7a29db7b-7128-47ba-8169-b2dd5882a1f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"
Sep 15 12:26:28.301874 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.301855    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gvl6\" (UniqueName: \"kubernetes.io/projected/49823d24-5865-4a85-bacb-195339de36b2-kube-api-access-5gvl6\") pod \"insights-operator-5844b856db-qccwt\" (UID: \"49823d24-5865-4a85-bacb-195339de36b2\") " pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.302156 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.302137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fjs4\" (UniqueName: \"kubernetes.io/projected/7a29db7b-7128-47ba-8169-b2dd5882a1f4-kube-api-access-5fjs4\") pod \"klusterlet-addon-workmgr-cc5977694-7sbbc\" (UID: \"7a29db7b-7128-47ba-8169-b2dd5882a1f4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"
Sep 15 12:26:28.385417 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.385384    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:28.394659 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.394633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-ca-trust-extracted\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.394769 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.394671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-image-registry-private-configuration\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.394769 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.394691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-registry-tls\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.394871 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.394853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-registry-certificates\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.394923 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.394911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-installation-pull-secrets\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.394972 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.394944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-trusted-ca\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.395028 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.394979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk684\" (UniqueName: \"kubernetes.io/projected/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-kube-api-access-bk684\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.395076 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.395038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-bound-sa-token\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.395130 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.395092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-ca-trust-extracted\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.395698 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.395681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-registry-certificates\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.396211 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.396189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-trusted-ca\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.397331 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.397309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-registry-tls\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.397416 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.397317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-installation-pull-secrets\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.397460 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.397414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-image-registry-private-configuration\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.405239 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.405211    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk684\" (UniqueName: \"kubernetes.io/projected/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-kube-api-access-bk684\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.405458 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.405441    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a90a1fc-7dd9-4350-9ead-8fce38cb37bf-bound-sa-token\") pod \"image-registry-79b48c6d55-27vqd\" (UID: \"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf\") " pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.421286 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.421268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-qccwt"
Sep 15 12:26:28.454360 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.454342    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"
Sep 15 12:26:28.483473 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.483440    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:28.510758 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.510709    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:28.724498 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.724428    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-qccwt"]
Sep 15 12:26:28.727563 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:26:28.727537    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49823d24_5865_4a85_bacb_195339de36b2.slice/crio-c81b647cc85baac8d42e3ae7ce6f72cff9822cca4988251e3b2427d685e6b414 WatchSource:0}: Error finding container c81b647cc85baac8d42e3ae7ce6f72cff9822cca4988251e3b2427d685e6b414: Status 404 returned error can't find the container with id c81b647cc85baac8d42e3ae7ce6f72cff9822cca4988251e3b2427d685e6b414
Sep 15 12:26:28.758354 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.758329    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"]
Sep 15 12:26:28.760780 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:26:28.760754    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a29db7b_7128_47ba_8169_b2dd5882a1f4.slice/crio-4474f2fd68849819435df9bf5dca916e811238ca9053da2b37a4cb4af2071df0 WatchSource:0}: Error finding container 4474f2fd68849819435df9bf5dca916e811238ca9053da2b37a4cb4af2071df0: Status 404 returned error can't find the container with id 4474f2fd68849819435df9bf5dca916e811238ca9053da2b37a4cb4af2071df0
Sep 15 12:26:28.813578 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:28.813549    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79b48c6d55-27vqd"]
Sep 15 12:26:28.816966 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:26:28.816938    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a90a1fc_7dd9_4350_9ead_8fce38cb37bf.slice/crio-b91cbc61bea5c06b5852cbc8cccf069ef67c4660d7fb6a5bfda08187243bb686 WatchSource:0}: Error finding container b91cbc61bea5c06b5852cbc8cccf069ef67c4660d7fb6a5bfda08187243bb686: Status 404 returned error can't find the container with id b91cbc61bea5c06b5852cbc8cccf069ef67c4660d7fb6a5bfda08187243bb686
Sep 15 12:26:29.074686 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:29.074609    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:29.178196 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:29.178165    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:29.374020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:29.373967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79b48c6d55-27vqd" event={"ID":"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf","Type":"ContainerStarted","Data":"473dc719550bc0deb518eceb80d3d74dfa6c6b56d4429d9810590b1f68e3938a"}
Sep 15 12:26:29.374201 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:29.374038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79b48c6d55-27vqd" event={"ID":"9a90a1fc-7dd9-4350-9ead-8fce38cb37bf","Type":"ContainerStarted","Data":"b91cbc61bea5c06b5852cbc8cccf069ef67c4660d7fb6a5bfda08187243bb686"}
Sep 15 12:26:29.374201 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:29.374107    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:29.375341 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:29.375305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc" event={"ID":"7a29db7b-7128-47ba-8169-b2dd5882a1f4","Type":"ContainerStarted","Data":"4474f2fd68849819435df9bf5dca916e811238ca9053da2b37a4cb4af2071df0"}
Sep 15 12:26:29.376517 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:29.376494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qccwt" event={"ID":"49823d24-5865-4a85-bacb-195339de36b2","Type":"ContainerStarted","Data":"c81b647cc85baac8d42e3ae7ce6f72cff9822cca4988251e3b2427d685e6b414"}
Sep 15 12:26:29.395845 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:29.395783    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79b48c6d55-27vqd" podStartSLOduration=1.3957673320000001 podStartE2EDuration="1.395767332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:29.395121135 +0000 UTC m=+60.779832020" watchObservedRunningTime="2026-09-15 12:26:29.395767332 +0000 UTC m=+60.780478219"
Sep 15 12:26:29.514638 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:29.514585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:29.622603 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:29.622568    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:30.270397 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:30.270369    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rpkkg"
Sep 15 12:26:30.422849 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:30.422801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:30.478856 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:30.478778    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:30.716010 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:30.715975    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:30.816960 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:30.816877    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:31.112559 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.112534    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5stdh"]
Sep 15 12:26:31.114657 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.114632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.117128 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.117103    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 12:26:31.117128 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.117119    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 12:26:31.117329 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.117147    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m9kbt\""
Sep 15 12:26:31.129398 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.129377    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5stdh"]
Sep 15 12:26:31.152396 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.152363    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:31.224022 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.223993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.224159 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.224036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.224159 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.224059    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a047387b-02b3-4291-bc1c-e61fedc7aea9-data-volume\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.224159 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.224078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a047387b-02b3-4291-bc1c-e61fedc7aea9-crio-socket\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.224159 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.224120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvs6h\" (UniqueName: \"kubernetes.io/projected/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-api-access-pvs6h\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.229907 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.229875    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:31.325149 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.325114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.325530 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.325159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.325530 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:31.325238    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 12:26:31.325530 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:31.325253    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 12:26:31.325530 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.325275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a047387b-02b3-4291-bc1c-e61fedc7aea9-data-volume\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.325530 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:31.325302    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls podName:a047387b-02b3-4291-bc1c-e61fedc7aea9 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:31.825288903 +0000 UTC m=+63.209999792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5stdh" (UID: "a047387b-02b3-4291-bc1c-e61fedc7aea9") : secret "insights-runtime-extractor-tls" not found
Sep 15 12:26:31.325530 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.325322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a047387b-02b3-4291-bc1c-e61fedc7aea9-crio-socket\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.325530 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:31.325340    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm podName:a047387b-02b3-4291-bc1c-e61fedc7aea9 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:31.825323113 +0000 UTC m=+63.210033984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5stdh" (UID: "a047387b-02b3-4291-bc1c-e61fedc7aea9") : configmap "kube-rbac-proxy" not found
Sep 15 12:26:31.325530 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.325370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a047387b-02b3-4291-bc1c-e61fedc7aea9-crio-socket\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.325530 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.325416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvs6h\" (UniqueName: \"kubernetes.io/projected/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-api-access-pvs6h\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.325826 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.325591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a047387b-02b3-4291-bc1c-e61fedc7aea9-data-volume\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.344700 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.344677    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvs6h\" (UniqueName: \"kubernetes.io/projected/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-api-access-pvs6h\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.351128 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.351099    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:31.385048 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.384975    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-qccwt" event={"ID":"49823d24-5865-4a85-bacb-195339de36b2","Type":"ContainerStarted","Data":"f617d5a0eb236232e48367fb021de6eab867c45bfaa807798e61154ef416947b"}
Sep 15 12:26:31.403773 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.403732    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-qccwt" podStartSLOduration=1.611873412 podStartE2EDuration="3.403717316s" totalImagesPullingTime="1.791843904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:31.402702971 +0000 UTC m=+62.787413854" watchObservedRunningTime="2026-09-15 12:26:31.403717316 +0000 UTC m=+62.788428231"
Sep 15 12:26:31.691578 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.691513    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2ndsn_7812e2fd-cb28-4988-a3e0-5d9433a853f1/dns/0.log"
Sep 15 12:26:31.831089 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.831055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.831258 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.831155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:31.831258 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:31.831214    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 12:26:31.831351 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:31.831266    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 12:26:31.831351 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:31.831277    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls podName:a047387b-02b3-4291-bc1c-e61fedc7aea9 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:32.831262097 +0000 UTC m=+64.215972963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5stdh" (UID: "a047387b-02b3-4291-bc1c-e61fedc7aea9") : secret "insights-runtime-extractor-tls" not found
Sep 15 12:26:31.831351 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:31.831328    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm podName:a047387b-02b3-4291-bc1c-e61fedc7aea9 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:32.831312678 +0000 UTC m=+64.216023541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5stdh" (UID: "a047387b-02b3-4291-bc1c-e61fedc7aea9") : configmap "kube-rbac-proxy" not found
Sep 15 12:26:31.891599 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:31.891573    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2ndsn_7812e2fd-cb28-4988-a3e0-5d9433a853f1/kube-rbac-proxy/0.log"
Sep 15 12:26:32.354754 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:32.354729    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2ndsn"
Sep 15 12:26:32.839416 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:32.839331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:32.839416 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:32.839404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:32.839730 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:32.839467    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 12:26:32.839730 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:32.839536    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 12:26:32.839730 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:32.839545    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm podName:a047387b-02b3-4291-bc1c-e61fedc7aea9 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:34.839527697 +0000 UTC m=+66.224238561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5stdh" (UID: "a047387b-02b3-4291-bc1c-e61fedc7aea9") : configmap "kube-rbac-proxy" not found
Sep 15 12:26:32.839730 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:32.839598    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls podName:a047387b-02b3-4291-bc1c-e61fedc7aea9 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:34.839582144 +0000 UTC m=+66.224293009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5stdh" (UID: "a047387b-02b3-4291-bc1c-e61fedc7aea9") : secret "insights-runtime-extractor-tls" not found
Sep 15 12:26:32.888947 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:32.888913    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:32.893152 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:32.893129    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-66lhz_866085d7-f0ed-4104-bd66-d5964b6619c9/dns-node-resolver/0.log"
Sep 15 12:26:32.990298 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:32.990232    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:33.086580 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:33.086537    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:33.187512 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:33.187464    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:33.292161 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:33.292131    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:33.370053 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:33.369967    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:33.492267 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:33.492189    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-57995fd64d-tj4cl_bce2551d-652b-4087-a003-003897e1c854/registry/0.log"
Sep 15 12:26:33.527253 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:33.527219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:33.891800 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:33.891758    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79b48c6d55-27vqd_9a90a1fc-7dd9-4350-9ead-8fce38cb37bf/registry/0.log"
Sep 15 12:26:34.091420 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.091392    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-blcfn_1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc/node-ca/0.log"
Sep 15 12:26:34.658073 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.658030    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:34.757361 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.757322    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:34.809448 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.809413    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:34.854244 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.854208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5w4p\" (UniqueName: \"kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p\") pod \"network-check-target-n4m28\" (UID: \"2b715742-769a-4263-a81d-afd9e2c29183\") " pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:26:34.854473 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.854264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:26:34.854473 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.854295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:34.854473 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.854332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:34.854649 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:34.854497    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 12:26:34.854649 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:34.854561    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls podName:a047387b-02b3-4291-bc1c-e61fedc7aea9 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:38.854541623 +0000 UTC m=+70.239252502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5stdh" (UID: "a047387b-02b3-4291-bc1c-e61fedc7aea9") : secret "insights-runtime-extractor-tls" not found
Sep 15 12:26:34.854649 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:34.854603    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 12:26:34.854789 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:34.854664    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm podName:a047387b-02b3-4291-bc1c-e61fedc7aea9 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:38.854645711 +0000 UTC m=+70.239356588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5stdh" (UID: "a047387b-02b3-4291-bc1c-e61fedc7aea9") : configmap "kube-rbac-proxy" not found
Sep 15 12:26:34.856848 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.856813    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 12:26:34.856980 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.856905    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 12:26:34.866928 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.866897    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 12:26:34.867317 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.867291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1-original-pull-secret\") pod \"global-pull-secret-syncer-6txlk\" (UID: \"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1\") " pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:26:34.878271 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.878248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5w4p\" (UniqueName: \"kubernetes.io/projected/2b715742-769a-4263-a81d-afd9e2c29183-kube-api-access-q5w4p\") pod \"network-check-target-n4m28\" (UID: \"2b715742-769a-4263-a81d-afd9e2c29183\") " pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:26:34.891717 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.891694    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bx6hs_40dd46b2-753c-40ea-a811-15944e4951b0/serve-healthcheck-canary/0.log"
Sep 15 12:26:34.955211 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.955131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs\") pod \"network-metrics-daemon-bsb55\" (UID: \"15a5076c-b542-4a0d-bad7-cc06b81bbe3e\") " pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:26:34.957595 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.957573    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 12:26:34.967969 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:34.967945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15a5076c-b542-4a0d-bad7-cc06b81bbe3e-metrics-certs\") pod \"network-metrics-daemon-bsb55\" (UID: \"15a5076c-b542-4a0d-bad7-cc06b81bbe3e\") " pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:26:35.114407 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:35.114375    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j4plp\""
Sep 15 12:26:35.119530 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:35.119508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6txlk"
Sep 15 12:26:35.122614 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:35.122595    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bsb55"
Sep 15 12:26:35.128288 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:35.128267    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mnpg9\""
Sep 15 12:26:35.136386 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:35.136365    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:26:35.919166 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:35.919130    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n4m28"]
Sep 15 12:26:35.922052 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:26:35.922021    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b715742_769a_4263_a81d_afd9e2c29183.slice/crio-ac12a4ccf63e36ff7d6f20292b22f09a8eccdaa9135c9f66b0dada6e4777633c WatchSource:0}: Error finding container ac12a4ccf63e36ff7d6f20292b22f09a8eccdaa9135c9f66b0dada6e4777633c: Status 404 returned error can't find the container with id ac12a4ccf63e36ff7d6f20292b22f09a8eccdaa9135c9f66b0dada6e4777633c
Sep 15 12:26:35.922991 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:35.922964    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bsb55"]
Sep 15 12:26:35.923936 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:35.923798    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6txlk"]
Sep 15 12:26:35.925974 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:26:35.925939    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a5076c_b542_4a0d_bad7_cc06b81bbe3e.slice/crio-3e8dde29369cc69927d556a617c193c8ae51aac6774c3091e8c6a97df62379d8 WatchSource:0}: Error finding container 3e8dde29369cc69927d556a617c193c8ae51aac6774c3091e8c6a97df62379d8: Status 404 returned error can't find the container with id 3e8dde29369cc69927d556a617c193c8ae51aac6774c3091e8c6a97df62379d8
Sep 15 12:26:35.927113 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:26:35.927078    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f04d5a_e442_4cf8_ba3a_1b37b7934fb1.slice/crio-65ed71184f56bb40574d5e0c12d8e5b65f84eaafe57c8a37f01e5e129bf203c9 WatchSource:0}: Error finding container 65ed71184f56bb40574d5e0c12d8e5b65f84eaafe57c8a37f01e5e129bf203c9: Status 404 returned error can't find the container with id 65ed71184f56bb40574d5e0c12d8e5b65f84eaafe57c8a37f01e5e129bf203c9
Sep 15 12:26:36.405620 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:36.405574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6txlk" event={"ID":"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1","Type":"ContainerStarted","Data":"65ed71184f56bb40574d5e0c12d8e5b65f84eaafe57c8a37f01e5e129bf203c9"}
Sep 15 12:26:36.407291 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:36.407259    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n4m28" event={"ID":"2b715742-769a-4263-a81d-afd9e2c29183","Type":"ContainerStarted","Data":"ac12a4ccf63e36ff7d6f20292b22f09a8eccdaa9135c9f66b0dada6e4777633c"}
Sep 15 12:26:36.408700 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:36.408674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bsb55" event={"ID":"15a5076c-b542-4a0d-bad7-cc06b81bbe3e","Type":"ContainerStarted","Data":"3e8dde29369cc69927d556a617c193c8ae51aac6774c3091e8c6a97df62379d8"}
Sep 15 12:26:36.410785 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:36.410714    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc" event={"ID":"7a29db7b-7128-47ba-8169-b2dd5882a1f4","Type":"ContainerStarted","Data":"a1fbc3c404927a267da7cb47e6a0bd20df50426321ec4657a5433ed89ab5c964"}
Sep 15 12:26:36.411186 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:36.411126    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"
Sep 15 12:26:36.413060 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:36.413025    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc"
Sep 15 12:26:36.426544 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:36.426496    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cc5977694-7sbbc" podStartSLOduration=1.4554743829999999 podStartE2EDuration="8.426479829s" totalImagesPullingTime="6.971005446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:36.425772105 +0000 UTC m=+67.810482989" watchObservedRunningTime="2026-09-15 12:26:36.426479829 +0000 UTC m=+67.811190715"
Sep 15 12:26:37.274851 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:37.274740    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:37.376272 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:37.376190    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:37.418477 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:37.418441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bsb55" event={"ID":"15a5076c-b542-4a0d-bad7-cc06b81bbe3e","Type":"ContainerStarted","Data":"4917d5f12b336be725d7cea62af45eba965f12b6935b58ab32986c184a04c7a0"}
Sep 15 12:26:37.418649 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:37.418510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bsb55" event={"ID":"15a5076c-b542-4a0d-bad7-cc06b81bbe3e","Type":"ContainerStarted","Data":"3efd52b138d8635934542ccaa7a58f9f3f08f9f1c6a638b866fed83f8aed91ec"}
Sep 15 12:26:37.436217 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:37.436005    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bsb55" podStartSLOduration=66.419592651 podStartE2EDuration="1m7.435984785s" totalImagesPullingTime="1.016392134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:37.434710046 +0000 UTC m=+68.819420932" watchObservedRunningTime="2026-09-15 12:26:37.435984785 +0000 UTC m=+68.820695668"
Sep 15 12:26:37.450189 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:37.450156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:37.541672 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:37.541593    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:38.144813 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:38.144768    2579 patch_prober.go:28] interesting pod/image-registry-57995fd64d-tj4cl container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 12:26:38.144992 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:38.144855    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-57995fd64d-tj4cl" podUID="bce2551d-652b-4087-a003-003897e1c854" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 12:26:38.888005 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:38.887965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:38.888450 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:38.888015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:38.888450 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:38.888097    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 12:26:38.888450 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:38.888144    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 12:26:38.888450 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:38.888168    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm podName:a047387b-02b3-4291-bc1c-e61fedc7aea9 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:46.888147421 +0000 UTC m=+78.272858301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5stdh" (UID: "a047387b-02b3-4291-bc1c-e61fedc7aea9") : configmap "kube-rbac-proxy" not found
Sep 15 12:26:38.888450 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:38.888185    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls podName:a047387b-02b3-4291-bc1c-e61fedc7aea9 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:46.888176677 +0000 UTC m=+78.272887548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5stdh" (UID: "a047387b-02b3-4291-bc1c-e61fedc7aea9") : secret "insights-runtime-extractor-tls" not found
Sep 15 12:26:40.312179 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:40.312115    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:40.412754 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:40.412715    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:41.434561 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:41.434521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6txlk" event={"ID":"f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1","Type":"ContainerStarted","Data":"d3f99d30368f22ddaeb0fdb81a12fd1b640ec1a0d25f46c8e9cc00c4a4652f9e"}
Sep 15 12:26:41.436068 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:41.436047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n4m28" event={"ID":"2b715742-769a-4263-a81d-afd9e2c29183","Type":"ContainerStarted","Data":"8948e430009410ae60d310c2a7adfcbeb8a6c49e6556d7d6adef46c144fde408"}
Sep 15 12:26:41.436192 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:41.436103    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:26:41.449029 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:41.448990    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6txlk" podStartSLOduration=66.341264823 podStartE2EDuration="1m11.448977945s" totalImagesPullingTime="5.107713122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:41.448533664 +0000 UTC m=+72.833244551" watchObservedRunningTime="2026-09-15 12:26:41.448977945 +0000 UTC m=+72.833688868"
Sep 15 12:26:41.467223 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:41.467187    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n4m28" podStartSLOduration=66.364834492 podStartE2EDuration="1m11.467174574s" totalImagesPullingTime="5.102340082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:41.466690969 +0000 UTC m=+72.851401854" watchObservedRunningTime="2026-09-15 12:26:41.467174574 +0000 UTC m=+72.851885458"
Sep 15 12:26:42.751790 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.751753    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"]
Sep 15 12:26:42.775983 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.775956    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-88w75"]
Sep 15 12:26:42.776144 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.776096    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"
Sep 15 12:26:42.778464 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.778437    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 12:26:42.778614 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.778473    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 12:26:42.779571 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.779550    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 15 12:26:42.779687 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.779557    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4xz6g\""
Sep 15 12:26:42.779740 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.779557    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 12:26:42.779918 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.779902    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 15 12:26:42.792003 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.791983    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"]
Sep 15 12:26:42.792104 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.792073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:42.794395 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.794378    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 12:26:42.794517 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.794449    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 12:26:42.794517 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.794474    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wzdhx\""
Sep 15 12:26:42.794812 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.794785    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 12:26:42.823841 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.823808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9de9d475-0445-47d4-9b02-5f92e1bc1a85-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-j7njs\" (UID: \"9de9d475-0445-47d4-9b02-5f92e1bc1a85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"
Sep 15 12:26:42.823955 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.823860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de9d475-0445-47d4-9b02-5f92e1bc1a85-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-j7njs\" (UID: \"9de9d475-0445-47d4-9b02-5f92e1bc1a85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"
Sep 15 12:26:42.823955 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.823882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9de9d475-0445-47d4-9b02-5f92e1bc1a85-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-j7njs\" (UID: \"9de9d475-0445-47d4-9b02-5f92e1bc1a85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"
Sep 15 12:26:42.824057 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.823972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpt45\" (UniqueName: \"kubernetes.io/projected/9de9d475-0445-47d4-9b02-5f92e1bc1a85-kube-api-access-bpt45\") pod \"openshift-state-metrics-67dcf8c979-j7njs\" (UID: \"9de9d475-0445-47d4-9b02-5f92e1bc1a85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"
Sep 15 12:26:42.924886 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.924851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-metrics-client-ca\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:42.924886 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.924887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgz2m\" (UniqueName: \"kubernetes.io/projected/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-kube-api-access-tgz2m\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:42.925122 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.924931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9de9d475-0445-47d4-9b02-5f92e1bc1a85-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-j7njs\" (UID: \"9de9d475-0445-47d4-9b02-5f92e1bc1a85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"
Sep 15 12:26:42.925122 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.924958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de9d475-0445-47d4-9b02-5f92e1bc1a85-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-j7njs\" (UID: \"9de9d475-0445-47d4-9b02-5f92e1bc1a85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"
Sep 15 12:26:42.925122 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.924984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-tls\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:42.925122 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.925014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9de9d475-0445-47d4-9b02-5f92e1bc1a85-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-j7njs\" (UID: \"9de9d475-0445-47d4-9b02-5f92e1bc1a85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"
Sep 15 12:26:42.925122 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.925041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-wtmp\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:42.925122 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.925100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:42.925410 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.925180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpt45\" (UniqueName: \"kubernetes.io/projected/9de9d475-0445-47d4-9b02-5f92e1bc1a85-kube-api-access-bpt45\") pod \"openshift-state-metrics-67dcf8c979-j7njs\" (UID: \"9de9d475-0445-47d4-9b02-5f92e1bc1a85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"
Sep 15 12:26:42.925410 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.925234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-root\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:42.925410 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.925264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-sys\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:42.925410 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.925310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-accelerators-collector-config\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:42.925410 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.925358    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-textfile\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:42.925843 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.925803    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9de9d475-0445-47d4-9b02-5f92e1bc1a85-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-j7njs\" (UID: \"9de9d475-0445-47d4-9b02-5f92e1bc1a85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"
Sep 15 12:26:42.927458 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.927429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9de9d475-0445-47d4-9b02-5f92e1bc1a85-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-j7njs\" (UID: \"9de9d475-0445-47d4-9b02-5f92e1bc1a85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"
Sep 15 12:26:42.927563 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.927513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de9d475-0445-47d4-9b02-5f92e1bc1a85-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-j7njs\" (UID: \"9de9d475-0445-47d4-9b02-5f92e1bc1a85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"
Sep 15 12:26:42.934120 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:42.934091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpt45\" (UniqueName: \"kubernetes.io/projected/9de9d475-0445-47d4-9b02-5f92e1bc1a85-kube-api-access-bpt45\") pod \"openshift-state-metrics-67dcf8c979-j7njs\" (UID: \"9de9d475-0445-47d4-9b02-5f92e1bc1a85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"
Sep 15 12:26:43.026581 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.026507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-metrics-client-ca\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.026581 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.026548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgz2m\" (UniqueName: \"kubernetes.io/projected/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-kube-api-access-tgz2m\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.026776 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.026601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-tls\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.026776 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.026637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-wtmp\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.026776 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.026681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.026776 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.026735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-root\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.026776 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:43.026742    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 15 12:26:43.026776 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.026771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-sys\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.027115 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:43.026803    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-tls podName:0aeb0879-d842-44e4-a4ec-4494c5d7e48d nodeName:}" failed. No retries permitted until 2026-09-15 12:26:43.526783415 +0000 UTC m=+74.911494288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-tls") pod "node-exporter-88w75" (UID: "0aeb0879-d842-44e4-a4ec-4494c5d7e48d") : secret "node-exporter-tls" not found
Sep 15 12:26:43.027115 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.026822    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-sys\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.027115 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.026867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-accelerators-collector-config\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.027115 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.026883    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-root\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.027115 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.026882    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-wtmp\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.027115 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.026903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-textfile\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.027340 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.027155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-metrics-client-ca\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.027340 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.027175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-textfile\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.027406 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.027340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-accelerators-collector-config\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.028896 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.028880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.036512 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.036489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgz2m\" (UniqueName: \"kubernetes.io/projected/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-kube-api-access-tgz2m\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.087084 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.087054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"
Sep 15 12:26:43.388995 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.388778    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs"]
Sep 15 12:26:43.391551 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:26:43.391522    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de9d475_0445_47d4_9b02_5f92e1bc1a85.slice/crio-3ffe370a61bd3784e9940d144b60bc123b65c5719d1e5b4cc5a918c87c4c8490 WatchSource:0}: Error finding container 3ffe370a61bd3784e9940d144b60bc123b65c5719d1e5b4cc5a918c87c4c8490: Status 404 returned error can't find the container with id 3ffe370a61bd3784e9940d144b60bc123b65c5719d1e5b4cc5a918c87c4c8490
Sep 15 12:26:43.426668 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.426638    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:43.443742 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.443719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs" event={"ID":"9de9d475-0445-47d4-9b02-5f92e1bc1a85","Type":"ContainerStarted","Data":"3ffe370a61bd3784e9940d144b60bc123b65c5719d1e5b4cc5a918c87c4c8490"}
Sep 15 12:26:43.526629 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.526600    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:43.531249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.531230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-tls\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.533545 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.533519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0aeb0879-d842-44e4-a4ec-4494c5d7e48d-node-exporter-tls\") pod \"node-exporter-88w75\" (UID: \"0aeb0879-d842-44e4-a4ec-4494c5d7e48d\") " pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.701285 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.701214    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-88w75"
Sep 15 12:26:43.719620 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:26:43.719590    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aeb0879_d842_44e4_a4ec_4494c5d7e48d.slice/crio-5d6fd73cd8745d78d6a10381f57cb0f9a38c2485ccaac15e9173c73e2522cee3 WatchSource:0}: Error finding container 5d6fd73cd8745d78d6a10381f57cb0f9a38c2485ccaac15e9173c73e2522cee3: Status 404 returned error can't find the container with id 5d6fd73cd8745d78d6a10381f57cb0f9a38c2485ccaac15e9173c73e2522cee3
Sep 15 12:26:43.772502 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.772459    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:43.844037 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:43.844001    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:44.079348 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:44.079264    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:44.197867 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:44.197817    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:44.450651 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:44.450618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-88w75" event={"ID":"0aeb0879-d842-44e4-a4ec-4494c5d7e48d","Type":"ContainerStarted","Data":"5d6fd73cd8745d78d6a10381f57cb0f9a38c2485ccaac15e9173c73e2522cee3"}
Sep 15 12:26:44.452693 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:44.452669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs" event={"ID":"9de9d475-0445-47d4-9b02-5f92e1bc1a85","Type":"ContainerStarted","Data":"a7eae45bc214c413f9cda8623b5769e802662604343020df1c3e437fb3da2174"}
Sep 15 12:26:44.452821 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:44.452718    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs" event={"ID":"9de9d475-0445-47d4-9b02-5f92e1bc1a85","Type":"ContainerStarted","Data":"9ea962d21a80c3b855f4fa7a844deeed2efeb0d55bb6598e8306a957e5d62156"}
Sep 15 12:26:44.680458 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:44.680397    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:44.778053 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:44.777970    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:45.458193 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.458161    2579 generic.go:356] "Generic (PLEG): container finished" podID="0aeb0879-d842-44e4-a4ec-4494c5d7e48d" containerID="194a065fd568ab181fe2c46cba91c42350a54504fcd5cc2100356366456835f3" exitCode=0
Sep 15 12:26:45.458371 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.458249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-88w75" event={"ID":"0aeb0879-d842-44e4-a4ec-4494c5d7e48d","Type":"ContainerDied","Data":"194a065fd568ab181fe2c46cba91c42350a54504fcd5cc2100356366456835f3"}
Sep 15 12:26:45.460522 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.460496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs" event={"ID":"9de9d475-0445-47d4-9b02-5f92e1bc1a85","Type":"ContainerStarted","Data":"a6fd8a67e2180d494640a93573635f41a3a64d807c61119779f334f430e82a7c"}
Sep 15 12:26:45.492593 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.492548    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-j7njs" podStartSLOduration=2.116364305 podStartE2EDuration="3.492531296s" totalImagesPullingTime="1.376166991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:45.492105295 +0000 UTC m=+76.876816178" watchObservedRunningTime="2026-09-15 12:26:45.492531296 +0000 UTC m=+76.877242180"
Sep 15 12:26:45.705272 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.705234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:45.775442 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.775356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:45.776059 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.776037    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-68d4868d9c-prhnf"]
Sep 15 12:26:45.779033 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.779015    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.781488 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.781469    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 15 12:26:45.781585 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.781501    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 15 12:26:45.781659 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.781591    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 15 12:26:45.781659 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.781613    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tgdcv\""
Sep 15 12:26:45.781761 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.781736    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 15 12:26:45.781916 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.781897    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 15 12:26:45.782019 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.781963    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4s2uvkitvicis\""
Sep 15 12:26:45.790438 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.790420    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68d4868d9c-prhnf"]
Sep 15 12:26:45.853791 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.853767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-grpc-tls\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.853929 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.853806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.853929 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.853865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cncz\" (UniqueName: \"kubernetes.io/projected/51582373-a74d-4cc0-893e-78ad4df5ec9e-kube-api-access-4cncz\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.853929 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.853921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-thanos-querier-tls\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.854038 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.853960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.854038 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.854023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.854106 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.854044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.854106 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.854068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51582373-a74d-4cc0-893e-78ad4df5ec9e-metrics-client-ca\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.955108 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.955075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-thanos-querier-tls\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.955108 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.955110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.955311 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.955138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.955311 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.955256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.955311 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.955305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51582373-a74d-4cc0-893e-78ad4df5ec9e-metrics-client-ca\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.955450 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.955394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-grpc-tls\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.955450 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.955424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.955559 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.955471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cncz\" (UniqueName: \"kubernetes.io/projected/51582373-a74d-4cc0-893e-78ad4df5ec9e-kube-api-access-4cncz\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.956163 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.956116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51582373-a74d-4cc0-893e-78ad4df5ec9e-metrics-client-ca\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.958282 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.958256    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-thanos-querier-tls\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.958366 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.958298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.958366 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.958313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.958441 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.958413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.958441 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.958427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.958546 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.958531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51582373-a74d-4cc0-893e-78ad4df5ec9e-secret-grpc-tls\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:45.963240 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:45.963223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cncz\" (UniqueName: \"kubernetes.io/projected/51582373-a74d-4cc0-893e-78ad4df5ec9e-kube-api-access-4cncz\") pod \"thanos-querier-68d4868d9c-prhnf\" (UID: \"51582373-a74d-4cc0-893e-78ad4df5ec9e\") " pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:46.088102 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:46.088015    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:46.166705 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:46.166673    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:46.359472 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:46.359448    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68d4868d9c-prhnf"]
Sep 15 12:26:46.361244 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:26:46.361223    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51582373_a74d_4cc0_893e_78ad4df5ec9e.slice/crio-0fc7b3435b6a19475f73ae57770c31d0f8aed1d1c5e13131318379a216d02937 WatchSource:0}: Error finding container 0fc7b3435b6a19475f73ae57770c31d0f8aed1d1c5e13131318379a216d02937: Status 404 returned error can't find the container with id 0fc7b3435b6a19475f73ae57770c31d0f8aed1d1c5e13131318379a216d02937
Sep 15 12:26:46.465643 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:46.465608    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf" event={"ID":"51582373-a74d-4cc0-893e-78ad4df5ec9e","Type":"ContainerStarted","Data":"0fc7b3435b6a19475f73ae57770c31d0f8aed1d1c5e13131318379a216d02937"}
Sep 15 12:26:46.467677 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:46.467648    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-88w75" event={"ID":"0aeb0879-d842-44e4-a4ec-4494c5d7e48d","Type":"ContainerStarted","Data":"0c7093bf4734e6988820723afad0db001c5da558df0a443d1f5571d4bba50700"}
Sep 15 12:26:46.467803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:46.467693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-88w75" event={"ID":"0aeb0879-d842-44e4-a4ec-4494c5d7e48d","Type":"ContainerStarted","Data":"80712835e80b527ded25dfd058d3c9d35444e7b7354eaad4dc69cd358895e70b"}
Sep 15 12:26:46.484837 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:46.484795    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-88w75" podStartSLOduration=3.192942569 podStartE2EDuration="4.484781994s" totalImagesPullingTime="1.23106113s" totalInitContainerRuntime="60.778295ms" isStatefulPod=true podCreationTimestamp="2026-09-15 12:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:46.484036464 +0000 UTC m=+77.868747347" watchObservedRunningTime="2026-09-15 12:26:46.484781994 +0000 UTC m=+77.869492878"
Sep 15 12:26:46.964300 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:46.964267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:46.964761 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:46.964323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:46.964925 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:46.964892    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a047387b-02b3-4291-bc1c-e61fedc7aea9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:46.966995 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:46.966973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a047387b-02b3-4291-bc1c-e61fedc7aea9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5stdh\" (UID: \"a047387b-02b3-4291-bc1c-e61fedc7aea9\") " pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:47.024269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:47.024241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5stdh"
Sep 15 12:26:47.303353 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:47.302942    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5stdh"]
Sep 15 12:26:47.305936 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:26:47.305910    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda047387b_02b3_4291_bc1c_e61fedc7aea9.slice/crio-9ae96d923755aab859d5d7fe0d28de63a87b54aa7456d63e5dad9c4eaf45c0d8 WatchSource:0}: Error finding container 9ae96d923755aab859d5d7fe0d28de63a87b54aa7456d63e5dad9c4eaf45c0d8: Status 404 returned error can't find the container with id 9ae96d923755aab859d5d7fe0d28de63a87b54aa7456d63e5dad9c4eaf45c0d8
Sep 15 12:26:47.474016 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:47.473973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5stdh" event={"ID":"a047387b-02b3-4291-bc1c-e61fedc7aea9","Type":"ContainerStarted","Data":"e60f019406d8a11ed27804dabdaed273d47875b9189e9fbca09bcdb2bc389caf"}
Sep 15 12:26:47.474206 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:47.474051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5stdh" event={"ID":"a047387b-02b3-4291-bc1c-e61fedc7aea9","Type":"ContainerStarted","Data":"9ae96d923755aab859d5d7fe0d28de63a87b54aa7456d63e5dad9c4eaf45c0d8"}
Sep 15 12:26:47.540750 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:47.540705    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:47.594848 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:47.594748    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:47.680993 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:47.680964    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:47.781401 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:47.781359    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:48.128904 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:48.128867    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:48.141441 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:48.141417    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:48.480237 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:48.480205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5stdh" event={"ID":"a047387b-02b3-4291-bc1c-e61fedc7aea9","Type":"ContainerStarted","Data":"f3a4e17adcaa1ab06c7515b2dabf5db2b67abcb3c7e385f80c4bbd5cd3d7e36e"}
Sep 15 12:26:48.482521 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:48.482493    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf" event={"ID":"51582373-a74d-4cc0-893e-78ad4df5ec9e","Type":"ContainerStarted","Data":"109f2815b8d02b031d71856a758d42f74ba34b082b5ca0b6b72f602eae63d5ec"}
Sep 15 12:26:48.482632 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:48.482554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf" event={"ID":"51582373-a74d-4cc0-893e-78ad4df5ec9e","Type":"ContainerStarted","Data":"e176df4aaa977d412a92b999d34c869fb0bf5617ade30783830f9511d9f98f27"}
Sep 15 12:26:48.482632 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:48.482585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf" event={"ID":"51582373-a74d-4cc0-893e-78ad4df5ec9e","Type":"ContainerStarted","Data":"292f414472b1623c655aff182c3beb8c26059fe13e3f93abaaf01e7cf40eb21d"}
Sep 15 12:26:48.979097 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:48.979061    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:49.073260 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:49.073208    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:49.489139 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:49.489095    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf" event={"ID":"51582373-a74d-4cc0-893e-78ad4df5ec9e","Type":"ContainerStarted","Data":"2b555394258b65bd6425d41860f085fb18e83562b8f6d43e91dbcfc9a7344d80"}
Sep 15 12:26:50.384207 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:50.384178    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79b48c6d55-27vqd"
Sep 15 12:26:50.494531 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:50.494492    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5stdh" event={"ID":"a047387b-02b3-4291-bc1c-e61fedc7aea9","Type":"ContainerStarted","Data":"825cdc30907d6ec5f6440b0a1be39f1b96e2c6bb8a3ab134fe75068b9f4434b6"}
Sep 15 12:26:50.497580 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:50.497550    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf" event={"ID":"51582373-a74d-4cc0-893e-78ad4df5ec9e","Type":"ContainerStarted","Data":"1db85ddcf0addb25aeb2cb19f6dec20c885980b956394f463e2fcb5c6f7140d1"}
Sep 15 12:26:50.497719 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:50.497617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf" event={"ID":"51582373-a74d-4cc0-893e-78ad4df5ec9e","Type":"ContainerStarted","Data":"5bb3febb042286fbc2cb53b6d668210132fb79600aa9b161e6e2aeb36c2154ba"}
Sep 15 12:26:50.497719 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:50.497667    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:50.499517 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:50.499497    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:50.511718 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:50.511679    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5stdh" podStartSLOduration=17.233852249 podStartE2EDuration="19.511664522s" totalImagesPullingTime="2.277812273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:26:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:50.511095902 +0000 UTC m=+81.895806787" watchObservedRunningTime="2026-09-15 12:26:50.511664522 +0000 UTC m=+81.896375407"
Sep 15 12:26:50.531134 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:50.531095    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf" podStartSLOduration=2.858353841 podStartE2EDuration="5.531080774s" totalImagesPullingTime="2.672726933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:50.529371529 +0000 UTC m=+81.914082407" watchObservedRunningTime="2026-09-15 12:26:50.531080774 +0000 UTC m=+81.915791657"
Sep 15 12:26:50.600612 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:50.600586    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:50.700355 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:50.700275    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:51.820564 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:51.820449    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:51.921369 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:51.921336    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:53.153975 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.153933    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57995fd64d-tj4cl" podUID="bce2551d-652b-4087-a003-003897e1c854" containerName="registry" containerID="cri-o://6313733ae2bcfc914ec9407925433afc113a33bbcb34fb0d0eb112577bd01b69" gracePeriod=30
Sep 15 12:26:53.390841 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.390800    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:53.509540 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.509460    2579 generic.go:356] "Generic (PLEG): container finished" podID="bce2551d-652b-4087-a003-003897e1c854" containerID="6313733ae2bcfc914ec9407925433afc113a33bbcb34fb0d0eb112577bd01b69" exitCode=0
Sep 15 12:26:53.509540 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.509508    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57995fd64d-tj4cl"
Sep 15 12:26:53.509737 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.509544    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57995fd64d-tj4cl" event={"ID":"bce2551d-652b-4087-a003-003897e1c854","Type":"ContainerDied","Data":"6313733ae2bcfc914ec9407925433afc113a33bbcb34fb0d0eb112577bd01b69"}
Sep 15 12:26:53.509737 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.509591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57995fd64d-tj4cl" event={"ID":"bce2551d-652b-4087-a003-003897e1c854","Type":"ContainerDied","Data":"43476a985addc4100eb26c9c774546cb6d7e0bd800f1d9573b429f98a3faaa9d"}
Sep 15 12:26:53.509737 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.509612    2579 scope.go:117] "RemoveContainer" containerID="6313733ae2bcfc914ec9407925433afc113a33bbcb34fb0d0eb112577bd01b69"
Sep 15 12:26:53.517396 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.517380    2579 scope.go:117] "RemoveContainer" containerID="6313733ae2bcfc914ec9407925433afc113a33bbcb34fb0d0eb112577bd01b69"
Sep 15 12:26:53.517655 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:26:53.517635    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6313733ae2bcfc914ec9407925433afc113a33bbcb34fb0d0eb112577bd01b69\": container with ID starting with 6313733ae2bcfc914ec9407925433afc113a33bbcb34fb0d0eb112577bd01b69 not found: ID does not exist" containerID="6313733ae2bcfc914ec9407925433afc113a33bbcb34fb0d0eb112577bd01b69"
Sep 15 12:26:53.517698 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.517663    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6313733ae2bcfc914ec9407925433afc113a33bbcb34fb0d0eb112577bd01b69"} err="failed to get container status \"6313733ae2bcfc914ec9407925433afc113a33bbcb34fb0d0eb112577bd01b69\": rpc error: code = NotFound desc = could not find container \"6313733ae2bcfc914ec9407925433afc113a33bbcb34fb0d0eb112577bd01b69\": container with ID starting with 6313733ae2bcfc914ec9407925433afc113a33bbcb34fb0d0eb112577bd01b69 not found: ID does not exist"
Sep 15 12:26:53.522900 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.522882    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bce2551d-652b-4087-a003-003897e1c854-registry-certificates\") pod \"bce2551d-652b-4087-a003-003897e1c854\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") "
Sep 15 12:26:53.522968 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.522911    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bce2551d-652b-4087-a003-003897e1c854-installation-pull-secrets\") pod \"bce2551d-652b-4087-a003-003897e1c854\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") "
Sep 15 12:26:53.522968 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.522934    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bce2551d-652b-4087-a003-003897e1c854-image-registry-private-configuration\") pod \"bce2551d-652b-4087-a003-003897e1c854\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") "
Sep 15 12:26:53.522968 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.522958    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bce2551d-652b-4087-a003-003897e1c854-ca-trust-extracted\") pod \"bce2551d-652b-4087-a003-003897e1c854\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") "
Sep 15 12:26:53.523100 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.523080    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls\") pod \"bce2551d-652b-4087-a003-003897e1c854\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") "
Sep 15 12:26:53.523178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.523162    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-bound-sa-token\") pod \"bce2551d-652b-4087-a003-003897e1c854\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") "
Sep 15 12:26:53.523251 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.523192    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fdf9\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-kube-api-access-4fdf9\") pod \"bce2551d-652b-4087-a003-003897e1c854\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") "
Sep 15 12:26:53.523251 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.523219    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bce2551d-652b-4087-a003-003897e1c854-trusted-ca\") pod \"bce2551d-652b-4087-a003-003897e1c854\" (UID: \"bce2551d-652b-4087-a003-003897e1c854\") "
Sep 15 12:26:53.523353 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.523321    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce2551d-652b-4087-a003-003897e1c854-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bce2551d-652b-4087-a003-003897e1c854" (UID: "bce2551d-652b-4087-a003-003897e1c854"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:26:53.523673 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.523650    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bce2551d-652b-4087-a003-003897e1c854-registry-certificates\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:26:53.523850 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.523701    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce2551d-652b-4087-a003-003897e1c854-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bce2551d-652b-4087-a003-003897e1c854" (UID: "bce2551d-652b-4087-a003-003897e1c854"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:26:53.525516 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.525461    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce2551d-652b-4087-a003-003897e1c854-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bce2551d-652b-4087-a003-003897e1c854" (UID: "bce2551d-652b-4087-a003-003897e1c854"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:26:53.525676 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.525653    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce2551d-652b-4087-a003-003897e1c854-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bce2551d-652b-4087-a003-003897e1c854" (UID: "bce2551d-652b-4087-a003-003897e1c854"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:26:53.525759 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.525697    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bce2551d-652b-4087-a003-003897e1c854" (UID: "bce2551d-652b-4087-a003-003897e1c854"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:26:53.525759 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.525738    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bce2551d-652b-4087-a003-003897e1c854" (UID: "bce2551d-652b-4087-a003-003897e1c854"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:26:53.525884 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.525819    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-kube-api-access-4fdf9" (OuterVolumeSpecName: "kube-api-access-4fdf9") pod "bce2551d-652b-4087-a003-003897e1c854" (UID: "bce2551d-652b-4087-a003-003897e1c854"). InnerVolumeSpecName "kube-api-access-4fdf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:26:53.532230 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.532205    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce2551d-652b-4087-a003-003897e1c854-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bce2551d-652b-4087-a003-003897e1c854" (UID: "bce2551d-652b-4087-a003-003897e1c854"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:26:53.624248 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.624217    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bce2551d-652b-4087-a003-003897e1c854-installation-pull-secrets\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:26:53.624248 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.624244    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bce2551d-652b-4087-a003-003897e1c854-image-registry-private-configuration\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:26:53.624248 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.624254    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bce2551d-652b-4087-a003-003897e1c854-ca-trust-extracted\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:26:53.624446 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.624265    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-registry-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:26:53.624446 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.624273    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-bound-sa-token\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:26:53.624446 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.624281    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fdf9\" (UniqueName: \"kubernetes.io/projected/bce2551d-652b-4087-a003-003897e1c854-kube-api-access-4fdf9\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:26:53.624446 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.624289    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bce2551d-652b-4087-a003-003897e1c854-trusted-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:26:53.829868 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.829824    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57995fd64d-tj4cl"]
Sep 15 12:26:53.832316 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:53.832293    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57995fd64d-tj4cl"]
Sep 15 12:26:55.104505 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:55.104475    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bce2551d-652b-4087-a003-003897e1c854" path="/var/lib/kubelet/pods/bce2551d-652b-4087-a003-003897e1c854/volumes"
Sep 15 12:26:56.506907 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:56.506879    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-68d4868d9c-prhnf"
Sep 15 12:26:57.065740 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:57.065705    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:57.161677 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:57.161640    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:26:59.925388 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:26:59.925351    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:02.542194 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:27:02.542152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:02.603445 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:27:02.603414    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:07.889102 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:27:07.889066    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:07.949223 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:27:07.949183    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:12.441753 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:27:12.441722    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n4m28"
Sep 15 12:27:18.851699 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:27:18.851659    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:27.029939 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:27:27.029898    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:27.129947 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:27:27.129908    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:37.093440 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:27:37.093400    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:37.197014 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:27:37.196984    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:27:37.261372 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:27:37.261338    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:01.400652 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:01.400604    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:01.500227 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:01.500192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:01.988878 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:01.988827    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:02.430045 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:02.430010    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:10.187212 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:10.187182    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:10.249168 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:10.249105    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:17.202376 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:17.202342    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:17.280890 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:17.280858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:29.120127 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:29.120044    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:29.220235 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:29.220197    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:32.284736 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:32.284699    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:32.344013 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:32.343973    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:32.398032 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:32.397992    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:42.042992 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:42.042954    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:45.043649 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:45.043603    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:28:46.587119 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:28:46.587079    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:00.440384 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:00.440344    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:31.423910 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:31.423874    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:31.515251 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:31.515212    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:32.106347 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.106318    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"]
Sep 15 12:29:32.106618 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.106606    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce2551d-652b-4087-a003-003897e1c854" containerName="registry"
Sep 15 12:29:32.106664 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.106619    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce2551d-652b-4087-a003-003897e1c854" containerName="registry"
Sep 15 12:29:32.106705 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.106666    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce2551d-652b-4087-a003-003897e1c854" containerName="registry"
Sep 15 12:29:32.109477 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.109461    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"
Sep 15 12:29:32.111924 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.111895    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 12:29:32.112051 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.111897    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 12:29:32.112051 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.111952    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mtgf6\""
Sep 15 12:29:32.116445 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.116413    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"]
Sep 15 12:29:32.186386 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.186353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmr5b\" (UniqueName: \"kubernetes.io/projected/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-kube-api-access-tmr5b\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp\" (UID: \"7af98ed3-c2c6-43a5-b33a-36b53427dd1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"
Sep 15 12:29:32.186553 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.186402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp\" (UID: \"7af98ed3-c2c6-43a5-b33a-36b53427dd1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"
Sep 15 12:29:32.186721 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.186698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp\" (UID: \"7af98ed3-c2c6-43a5-b33a-36b53427dd1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"
Sep 15 12:29:32.287537 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.287510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp\" (UID: \"7af98ed3-c2c6-43a5-b33a-36b53427dd1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"
Sep 15 12:29:32.287680 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.287550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmr5b\" (UniqueName: \"kubernetes.io/projected/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-kube-api-access-tmr5b\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp\" (UID: \"7af98ed3-c2c6-43a5-b33a-36b53427dd1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"
Sep 15 12:29:32.287680 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.287570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp\" (UID: \"7af98ed3-c2c6-43a5-b33a-36b53427dd1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"
Sep 15 12:29:32.287975 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.287955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp\" (UID: \"7af98ed3-c2c6-43a5-b33a-36b53427dd1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"
Sep 15 12:29:32.287975 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.287970    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp\" (UID: \"7af98ed3-c2c6-43a5-b33a-36b53427dd1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"
Sep 15 12:29:32.295667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.295650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmr5b\" (UniqueName: \"kubernetes.io/projected/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-kube-api-access-tmr5b\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp\" (UID: \"7af98ed3-c2c6-43a5-b33a-36b53427dd1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"
Sep 15 12:29:32.418943 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.418865    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"
Sep 15 12:29:32.685260 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:32.685174    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"]
Sep 15 12:29:32.689503 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:29:32.689477    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af98ed3_c2c6_43a5_b33a_36b53427dd1e.slice/crio-55942b68ef2c872d9f4f35aed2205ac08243d0318def6163bdd54f2a8751ea70 WatchSource:0}: Error finding container 55942b68ef2c872d9f4f35aed2205ac08243d0318def6163bdd54f2a8751ea70: Status 404 returned error can't find the container with id 55942b68ef2c872d9f4f35aed2205ac08243d0318def6163bdd54f2a8751ea70
Sep 15 12:29:33.062775 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:33.062681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp" event={"ID":"7af98ed3-c2c6-43a5-b33a-36b53427dd1e","Type":"ContainerStarted","Data":"55942b68ef2c872d9f4f35aed2205ac08243d0318def6163bdd54f2a8751ea70"}
Sep 15 12:29:39.087889 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:39.087854    2579 generic.go:356] "Generic (PLEG): container finished" podID="7af98ed3-c2c6-43a5-b33a-36b53427dd1e" containerID="043de5250d3e255e79e5d5ddd6de0587845e1585809d341ef61d9d68633073e3" exitCode=0
Sep 15 12:29:39.088296 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:39.087900    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp" event={"ID":"7af98ed3-c2c6-43a5-b33a-36b53427dd1e","Type":"ContainerDied","Data":"043de5250d3e255e79e5d5ddd6de0587845e1585809d341ef61d9d68633073e3"}
Sep 15 12:29:42.101279 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:42.101246    2579 generic.go:356] "Generic (PLEG): container finished" podID="7af98ed3-c2c6-43a5-b33a-36b53427dd1e" containerID="c51b21648f5f785b9094f24a6be08b730f0a850963b9ff2cc9628bdabe960e5b" exitCode=0
Sep 15 12:29:42.101695 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:42.101317    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp" event={"ID":"7af98ed3-c2c6-43a5-b33a-36b53427dd1e","Type":"ContainerDied","Data":"c51b21648f5f785b9094f24a6be08b730f0a850963b9ff2cc9628bdabe960e5b"}
Sep 15 12:29:50.133596 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:50.133558    2579 generic.go:356] "Generic (PLEG): container finished" podID="7af98ed3-c2c6-43a5-b33a-36b53427dd1e" containerID="c2cfb75beb477d6baccea62777db9e610dc2261610b7d3b48637c61926731cf7" exitCode=0
Sep 15 12:29:50.134010 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:50.133632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp" event={"ID":"7af98ed3-c2c6-43a5-b33a-36b53427dd1e","Type":"ContainerDied","Data":"c2cfb75beb477d6baccea62777db9e610dc2261610b7d3b48637c61926731cf7"}
Sep 15 12:29:51.256246 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:51.256221    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"
Sep 15 12:29:51.339755 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:51.339726    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-util\") pod \"7af98ed3-c2c6-43a5-b33a-36b53427dd1e\" (UID: \"7af98ed3-c2c6-43a5-b33a-36b53427dd1e\") "
Sep 15 12:29:51.339915 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:51.339764    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-bundle\") pod \"7af98ed3-c2c6-43a5-b33a-36b53427dd1e\" (UID: \"7af98ed3-c2c6-43a5-b33a-36b53427dd1e\") "
Sep 15 12:29:51.339915 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:51.339809    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmr5b\" (UniqueName: \"kubernetes.io/projected/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-kube-api-access-tmr5b\") pod \"7af98ed3-c2c6-43a5-b33a-36b53427dd1e\" (UID: \"7af98ed3-c2c6-43a5-b33a-36b53427dd1e\") "
Sep 15 12:29:51.340495 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:51.340464    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-bundle" (OuterVolumeSpecName: "bundle") pod "7af98ed3-c2c6-43a5-b33a-36b53427dd1e" (UID: "7af98ed3-c2c6-43a5-b33a-36b53427dd1e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:29:51.341948 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:51.341927    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-kube-api-access-tmr5b" (OuterVolumeSpecName: "kube-api-access-tmr5b") pod "7af98ed3-c2c6-43a5-b33a-36b53427dd1e" (UID: "7af98ed3-c2c6-43a5-b33a-36b53427dd1e"). InnerVolumeSpecName "kube-api-access-tmr5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:29:51.344631 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:51.344608    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-util" (OuterVolumeSpecName: "util") pod "7af98ed3-c2c6-43a5-b33a-36b53427dd1e" (UID: "7af98ed3-c2c6-43a5-b33a-36b53427dd1e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:29:51.440467 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:51.440393    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-util\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:29:51.440467 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:51.440419    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-bundle\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:29:51.440467 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:51.440428    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmr5b\" (UniqueName: \"kubernetes.io/projected/7af98ed3-c2c6-43a5-b33a-36b53427dd1e-kube-api-access-tmr5b\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:29:52.141905 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:52.141876    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp"
Sep 15 12:29:52.142074 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:52.141872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rs6hp" event={"ID":"7af98ed3-c2c6-43a5-b33a-36b53427dd1e","Type":"ContainerDied","Data":"55942b68ef2c872d9f4f35aed2205ac08243d0318def6163bdd54f2a8751ea70"}
Sep 15 12:29:52.142074 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:52.141989    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55942b68ef2c872d9f4f35aed2205ac08243d0318def6163bdd54f2a8751ea70"
Sep 15 12:29:57.952076 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:57.952042    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:29:59.310425 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:29:59.310382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:01.655153 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:01.655119    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:01.755461 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:01.755421    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:03.738683 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:03.738643    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:03.856164 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:03.856128    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:04.042199 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.042102    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:04.594338 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.594304    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"]
Sep 15 12:30:04.594697 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.594678    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af98ed3-c2c6-43a5-b33a-36b53427dd1e" containerName="extract"
Sep 15 12:30:04.594782 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.594699    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af98ed3-c2c6-43a5-b33a-36b53427dd1e" containerName="extract"
Sep 15 12:30:04.594782 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.594720    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af98ed3-c2c6-43a5-b33a-36b53427dd1e" containerName="pull"
Sep 15 12:30:04.594782 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.594727    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af98ed3-c2c6-43a5-b33a-36b53427dd1e" containerName="pull"
Sep 15 12:30:04.594782 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.594738    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af98ed3-c2c6-43a5-b33a-36b53427dd1e" containerName="util"
Sep 15 12:30:04.594782 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.594747    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af98ed3-c2c6-43a5-b33a-36b53427dd1e" containerName="util"
Sep 15 12:30:04.595042 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.594816    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7af98ed3-c2c6-43a5-b33a-36b53427dd1e" containerName="extract"
Sep 15 12:30:04.599191 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.599170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"
Sep 15 12:30:04.603072 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.603048    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mtgf6\""
Sep 15 12:30:04.603161 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.603085    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 12:30:04.603653 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.603640    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 12:30:04.615446 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.615424    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"]
Sep 15 12:30:04.620860 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.620814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ffd8e449-2e09-4821-89d6-f36644d4fc66-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks\" (UID: \"ffd8e449-2e09-4821-89d6-f36644d4fc66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"
Sep 15 12:30:04.620936 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.620883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shgnz\" (UniqueName: \"kubernetes.io/projected/ffd8e449-2e09-4821-89d6-f36644d4fc66-kube-api-access-shgnz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks\" (UID: \"ffd8e449-2e09-4821-89d6-f36644d4fc66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"
Sep 15 12:30:04.620988 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.620936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ffd8e449-2e09-4821-89d6-f36644d4fc66-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks\" (UID: \"ffd8e449-2e09-4821-89d6-f36644d4fc66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"
Sep 15 12:30:04.722001 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.721970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ffd8e449-2e09-4821-89d6-f36644d4fc66-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks\" (UID: \"ffd8e449-2e09-4821-89d6-f36644d4fc66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"
Sep 15 12:30:04.722147 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.722009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shgnz\" (UniqueName: \"kubernetes.io/projected/ffd8e449-2e09-4821-89d6-f36644d4fc66-kube-api-access-shgnz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks\" (UID: \"ffd8e449-2e09-4821-89d6-f36644d4fc66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"
Sep 15 12:30:04.722147 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.722047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ffd8e449-2e09-4821-89d6-f36644d4fc66-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks\" (UID: \"ffd8e449-2e09-4821-89d6-f36644d4fc66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"
Sep 15 12:30:04.722350 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.722331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ffd8e449-2e09-4821-89d6-f36644d4fc66-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks\" (UID: \"ffd8e449-2e09-4821-89d6-f36644d4fc66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"
Sep 15 12:30:04.722386 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.722368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ffd8e449-2e09-4821-89d6-f36644d4fc66-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks\" (UID: \"ffd8e449-2e09-4821-89d6-f36644d4fc66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"
Sep 15 12:30:04.734327 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.734302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shgnz\" (UniqueName: \"kubernetes.io/projected/ffd8e449-2e09-4821-89d6-f36644d4fc66-kube-api-access-shgnz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks\" (UID: \"ffd8e449-2e09-4821-89d6-f36644d4fc66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"
Sep 15 12:30:04.907982 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:04.907947    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"
Sep 15 12:30:05.177589 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:05.177570    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"]
Sep 15 12:30:05.179959 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:30:05.179935    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd8e449_2e09_4821_89d6_f36644d4fc66.slice/crio-8f32bfb8d122ad859c39fa3da4529c006d365be347bf40523829343167204da5 WatchSource:0}: Error finding container 8f32bfb8d122ad859c39fa3da4529c006d365be347bf40523829343167204da5: Status 404 returned error can't find the container with id 8f32bfb8d122ad859c39fa3da4529c006d365be347bf40523829343167204da5
Sep 15 12:30:05.186127 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:05.186101    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks" event={"ID":"ffd8e449-2e09-4821-89d6-f36644d4fc66","Type":"ContainerStarted","Data":"8f32bfb8d122ad859c39fa3da4529c006d365be347bf40523829343167204da5"}
Sep 15 12:30:06.139042 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.139009    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:06.191237 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.191202    2579 generic.go:356] "Generic (PLEG): container finished" podID="ffd8e449-2e09-4821-89d6-f36644d4fc66" containerID="cff273f6e5c879aa2bdce982a7fc2d763caf68f6bb2352e28e4e8484959c23fd" exitCode=0
Sep 15 12:30:06.191389 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.191287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks" event={"ID":"ffd8e449-2e09-4821-89d6-f36644d4fc66","Type":"ContainerDied","Data":"cff273f6e5c879aa2bdce982a7fc2d763caf68f6bb2352e28e4e8484959c23fd"}
Sep 15 12:30:06.542245 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.542167    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:06.565857 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.565803    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dk2p2"]
Sep 15 12:30:06.567893 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.567872    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dk2p2"
Sep 15 12:30:06.570286 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.570266    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 12:30:06.570415 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.570271    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 12:30:06.570415 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.570271    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4hz57\""
Sep 15 12:30:06.578922 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.578901    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dk2p2"]
Sep 15 12:30:06.637327 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.637284    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1e14a3b-e824-41c1-b0a1-fdf8b152aa3a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dk2p2\" (UID: \"e1e14a3b-e824-41c1-b0a1-fdf8b152aa3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dk2p2"
Sep 15 12:30:06.637496 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.637346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd6jn\" (UniqueName: \"kubernetes.io/projected/e1e14a3b-e824-41c1-b0a1-fdf8b152aa3a-kube-api-access-dd6jn\") pod \"cert-manager-cainjector-559cdd8c87-dk2p2\" (UID: \"e1e14a3b-e824-41c1-b0a1-fdf8b152aa3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dk2p2"
Sep 15 12:30:06.738090 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.738037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1e14a3b-e824-41c1-b0a1-fdf8b152aa3a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dk2p2\" (UID: \"e1e14a3b-e824-41c1-b0a1-fdf8b152aa3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dk2p2"
Sep 15 12:30:06.738285 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.738137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd6jn\" (UniqueName: \"kubernetes.io/projected/e1e14a3b-e824-41c1-b0a1-fdf8b152aa3a-kube-api-access-dd6jn\") pod \"cert-manager-cainjector-559cdd8c87-dk2p2\" (UID: \"e1e14a3b-e824-41c1-b0a1-fdf8b152aa3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dk2p2"
Sep 15 12:30:06.745932 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.745907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd6jn\" (UniqueName: \"kubernetes.io/projected/e1e14a3b-e824-41c1-b0a1-fdf8b152aa3a-kube-api-access-dd6jn\") pod \"cert-manager-cainjector-559cdd8c87-dk2p2\" (UID: \"e1e14a3b-e824-41c1-b0a1-fdf8b152aa3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dk2p2"
Sep 15 12:30:06.745932 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.745918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1e14a3b-e824-41c1-b0a1-fdf8b152aa3a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dk2p2\" (UID: \"e1e14a3b-e824-41c1-b0a1-fdf8b152aa3a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dk2p2"
Sep 15 12:30:06.887243 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:06.887204    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dk2p2"
Sep 15 12:30:07.156170 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:07.156099    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dk2p2"]
Sep 15 12:30:07.159760 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:30:07.159734    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1e14a3b_e824_41c1_b0a1_fdf8b152aa3a.slice/crio-23b2ce3cfdfe86a77d828db0f7c51bf746a8d6dd7076846b2ac74a19b96cb34f WatchSource:0}: Error finding container 23b2ce3cfdfe86a77d828db0f7c51bf746a8d6dd7076846b2ac74a19b96cb34f: Status 404 returned error can't find the container with id 23b2ce3cfdfe86a77d828db0f7c51bf746a8d6dd7076846b2ac74a19b96cb34f
Sep 15 12:30:07.195653 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:07.195620    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dk2p2" event={"ID":"e1e14a3b-e824-41c1-b0a1-fdf8b152aa3a","Type":"ContainerStarted","Data":"23b2ce3cfdfe86a77d828db0f7c51bf746a8d6dd7076846b2ac74a19b96cb34f"}
Sep 15 12:30:08.348672 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:08.348635    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:08.403850 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:08.403795    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:13.338616 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:13.338577    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:29.044012 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:29.043983    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:30:29.052940 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:29.052917    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 12:30:29.057346 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:29.057317    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:30:40.218757 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:40.218715    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:40.314216 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:40.314182    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dk2p2" event={"ID":"e1e14a3b-e824-41c1-b0a1-fdf8b152aa3a","Type":"ContainerStarted","Data":"940258d376bdbf71586db48860e7a985feafc36c2f1ff14f1cdc930d63a488fe"}
Sep 15 12:30:40.316031 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:40.316002    2579 generic.go:356] "Generic (PLEG): container finished" podID="ffd8e449-2e09-4821-89d6-f36644d4fc66" containerID="4c468fed4080d76b9c84ea082ae7487df026eb6ec07646c1f34170656fdfb31d" exitCode=0
Sep 15 12:30:40.316151 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:40.316052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks" event={"ID":"ffd8e449-2e09-4821-89d6-f36644d4fc66","Type":"ContainerDied","Data":"4c468fed4080d76b9c84ea082ae7487df026eb6ec07646c1f34170656fdfb31d"}
Sep 15 12:30:40.319017 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:40.318993    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:40.330737 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:40.330697    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dk2p2" podStartSLOduration=2.086319431 podStartE2EDuration="34.330684222s" totalImagesPullingTime="32.244364791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:30:40.328657891 +0000 UTC m=+311.713368776" watchObservedRunningTime="2026-09-15 12:30:40.330684222 +0000 UTC m=+311.715395106"
Sep 15 12:30:41.321809 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:41.321773    2579 generic.go:356] "Generic (PLEG): container finished" podID="ffd8e449-2e09-4821-89d6-f36644d4fc66" containerID="2bc569439f5d8dac0e4632086040a6913ce99be52ccd7f71ff1cee02e17354a1" exitCode=0
Sep 15 12:30:41.322213 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:41.321871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks" event={"ID":"ffd8e449-2e09-4821-89d6-f36644d4fc66","Type":"ContainerDied","Data":"2bc569439f5d8dac0e4632086040a6913ce99be52ccd7f71ff1cee02e17354a1"}
Sep 15 12:30:42.442521 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:42.442495    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"
Sep 15 12:30:42.514492 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:42.514463    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shgnz\" (UniqueName: \"kubernetes.io/projected/ffd8e449-2e09-4821-89d6-f36644d4fc66-kube-api-access-shgnz\") pod \"ffd8e449-2e09-4821-89d6-f36644d4fc66\" (UID: \"ffd8e449-2e09-4821-89d6-f36644d4fc66\") "
Sep 15 12:30:42.514675 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:42.514544    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ffd8e449-2e09-4821-89d6-f36644d4fc66-util\") pod \"ffd8e449-2e09-4821-89d6-f36644d4fc66\" (UID: \"ffd8e449-2e09-4821-89d6-f36644d4fc66\") "
Sep 15 12:30:42.514675 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:42.514578    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ffd8e449-2e09-4821-89d6-f36644d4fc66-bundle\") pod \"ffd8e449-2e09-4821-89d6-f36644d4fc66\" (UID: \"ffd8e449-2e09-4821-89d6-f36644d4fc66\") "
Sep 15 12:30:42.514986 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:42.514963    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd8e449-2e09-4821-89d6-f36644d4fc66-bundle" (OuterVolumeSpecName: "bundle") pod "ffd8e449-2e09-4821-89d6-f36644d4fc66" (UID: "ffd8e449-2e09-4821-89d6-f36644d4fc66"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:30:42.516541 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:42.516503    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd8e449-2e09-4821-89d6-f36644d4fc66-kube-api-access-shgnz" (OuterVolumeSpecName: "kube-api-access-shgnz") pod "ffd8e449-2e09-4821-89d6-f36644d4fc66" (UID: "ffd8e449-2e09-4821-89d6-f36644d4fc66"). InnerVolumeSpecName "kube-api-access-shgnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:30:42.519193 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:42.519172    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd8e449-2e09-4821-89d6-f36644d4fc66-util" (OuterVolumeSpecName: "util") pod "ffd8e449-2e09-4821-89d6-f36644d4fc66" (UID: "ffd8e449-2e09-4821-89d6-f36644d4fc66"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:30:42.616024 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:42.615992    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ffd8e449-2e09-4821-89d6-f36644d4fc66-util\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:30:42.616024 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:42.616023    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ffd8e449-2e09-4821-89d6-f36644d4fc66-bundle\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:30:42.616196 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:42.616036    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shgnz\" (UniqueName: \"kubernetes.io/projected/ffd8e449-2e09-4821-89d6-f36644d4fc66-kube-api-access-shgnz\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:30:43.335153 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:43.335121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks" event={"ID":"ffd8e449-2e09-4821-89d6-f36644d4fc66","Type":"ContainerDied","Data":"8f32bfb8d122ad859c39fa3da4529c006d365be347bf40523829343167204da5"}
Sep 15 12:30:43.335308 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:43.335191    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f32bfb8d122ad859c39fa3da4529c006d365be347bf40523829343167204da5"
Sep 15 12:30:43.335308 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:43.335131    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4jrks"
Sep 15 12:30:47.411468 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:47.411431    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:47.518987 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:47.518948    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:49.249512 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.249477    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2kf95"]
Sep 15 12:30:49.249906 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.249743    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd8e449-2e09-4821-89d6-f36644d4fc66" containerName="pull"
Sep 15 12:30:49.249906 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.249753    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd8e449-2e09-4821-89d6-f36644d4fc66" containerName="pull"
Sep 15 12:30:49.249906 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.249761    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd8e449-2e09-4821-89d6-f36644d4fc66" containerName="extract"
Sep 15 12:30:49.249906 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.249767    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd8e449-2e09-4821-89d6-f36644d4fc66" containerName="extract"
Sep 15 12:30:49.249906 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.249782    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd8e449-2e09-4821-89d6-f36644d4fc66" containerName="util"
Sep 15 12:30:49.249906 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.249788    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd8e449-2e09-4821-89d6-f36644d4fc66" containerName="util"
Sep 15 12:30:49.249906 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.249847    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffd8e449-2e09-4821-89d6-f36644d4fc66" containerName="extract"
Sep 15 12:30:49.252776 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.252761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2kf95"
Sep 15 12:30:49.255295 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.255274    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 12:30:49.256189 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.256173    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 12:30:49.256281 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.256183    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-kk2jr\""
Sep 15 12:30:49.260908 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.260889    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2kf95"]
Sep 15 12:30:49.370121 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.370083    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5066a2c0-78da-4d6e-ad65-b97806371ff6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2kf95\" (UID: \"5066a2c0-78da-4d6e-ad65-b97806371ff6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2kf95"
Sep 15 12:30:49.370290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.370130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfj94\" (UniqueName: \"kubernetes.io/projected/5066a2c0-78da-4d6e-ad65-b97806371ff6-kube-api-access-rfj94\") pod \"openshift-lws-operator-fd8ccff4c-2kf95\" (UID: \"5066a2c0-78da-4d6e-ad65-b97806371ff6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2kf95"
Sep 15 12:30:49.470767 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.470720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfj94\" (UniqueName: \"kubernetes.io/projected/5066a2c0-78da-4d6e-ad65-b97806371ff6-kube-api-access-rfj94\") pod \"openshift-lws-operator-fd8ccff4c-2kf95\" (UID: \"5066a2c0-78da-4d6e-ad65-b97806371ff6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2kf95"
Sep 15 12:30:49.470985 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.470825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5066a2c0-78da-4d6e-ad65-b97806371ff6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2kf95\" (UID: \"5066a2c0-78da-4d6e-ad65-b97806371ff6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2kf95"
Sep 15 12:30:49.471223 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.471201    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5066a2c0-78da-4d6e-ad65-b97806371ff6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2kf95\" (UID: \"5066a2c0-78da-4d6e-ad65-b97806371ff6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2kf95"
Sep 15 12:30:49.478748 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.478720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfj94\" (UniqueName: \"kubernetes.io/projected/5066a2c0-78da-4d6e-ad65-b97806371ff6-kube-api-access-rfj94\") pod \"openshift-lws-operator-fd8ccff4c-2kf95\" (UID: \"5066a2c0-78da-4d6e-ad65-b97806371ff6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2kf95"
Sep 15 12:30:49.563022 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.562939    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2kf95"
Sep 15 12:30:49.830698 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.830670    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2kf95"]
Sep 15 12:30:49.832996 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:30:49.832969    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5066a2c0_78da_4d6e_ad65_b97806371ff6.slice/crio-d5bf3b07252236a7993faf35a623d6eb5972d461d8f6e244e35847700f4eaeff WatchSource:0}: Error finding container d5bf3b07252236a7993faf35a623d6eb5972d461d8f6e244e35847700f4eaeff: Status 404 returned error can't find the container with id d5bf3b07252236a7993faf35a623d6eb5972d461d8f6e244e35847700f4eaeff
Sep 15 12:30:49.834469 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:49.834448    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:30:50.364090 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:50.364053    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2kf95" event={"ID":"5066a2c0-78da-4d6e-ad65-b97806371ff6","Type":"ContainerStarted","Data":"d5bf3b07252236a7993faf35a623d6eb5972d461d8f6e244e35847700f4eaeff"}
Sep 15 12:30:53.378512 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:53.378479    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2kf95" event={"ID":"5066a2c0-78da-4d6e-ad65-b97806371ff6","Type":"ContainerStarted","Data":"954093cc047d81abf8b6c5583946dead7436706b394e7132a27972241580fe33"}
Sep 15 12:30:53.395404 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:53.395352    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2kf95" podStartSLOduration=1.901172158 podStartE2EDuration="4.395335499s" totalImagesPullingTime="2.494163341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:30:53.393723844 +0000 UTC m=+324.778434729" watchObservedRunningTime="2026-09-15 12:30:53.395335499 +0000 UTC m=+324.780046383"
Sep 15 12:30:54.135889 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:54.135823    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:54.229049 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:54.229011    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:55.573605 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.573570    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"]
Sep 15 12:30:55.577082 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.577064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"
Sep 15 12:30:55.579565 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.579544    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 12:30:55.579665 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.579546    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 12:30:55.579665 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.579578    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mtgf6\""
Sep 15 12:30:55.584043 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.584011    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"]
Sep 15 12:30:55.721253 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.721213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60c62d6a-e1ab-4d63-957b-346b65ec2263-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf\" (UID: \"60c62d6a-e1ab-4d63-957b-346b65ec2263\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"
Sep 15 12:30:55.721435 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.721258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60c62d6a-e1ab-4d63-957b-346b65ec2263-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf\" (UID: \"60c62d6a-e1ab-4d63-957b-346b65ec2263\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"
Sep 15 12:30:55.721435 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.721293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z77rv\" (UniqueName: \"kubernetes.io/projected/60c62d6a-e1ab-4d63-957b-346b65ec2263-kube-api-access-z77rv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf\" (UID: \"60c62d6a-e1ab-4d63-957b-346b65ec2263\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"
Sep 15 12:30:55.822542 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.822507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z77rv\" (UniqueName: \"kubernetes.io/projected/60c62d6a-e1ab-4d63-957b-346b65ec2263-kube-api-access-z77rv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf\" (UID: \"60c62d6a-e1ab-4d63-957b-346b65ec2263\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"
Sep 15 12:30:55.822691 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.822585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60c62d6a-e1ab-4d63-957b-346b65ec2263-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf\" (UID: \"60c62d6a-e1ab-4d63-957b-346b65ec2263\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"
Sep 15 12:30:55.822691 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.822605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60c62d6a-e1ab-4d63-957b-346b65ec2263-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf\" (UID: \"60c62d6a-e1ab-4d63-957b-346b65ec2263\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"
Sep 15 12:30:55.822971 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.822952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60c62d6a-e1ab-4d63-957b-346b65ec2263-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf\" (UID: \"60c62d6a-e1ab-4d63-957b-346b65ec2263\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"
Sep 15 12:30:55.823057 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.822985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60c62d6a-e1ab-4d63-957b-346b65ec2263-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf\" (UID: \"60c62d6a-e1ab-4d63-957b-346b65ec2263\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"
Sep 15 12:30:55.830226 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.830173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z77rv\" (UniqueName: \"kubernetes.io/projected/60c62d6a-e1ab-4d63-957b-346b65ec2263-kube-api-access-z77rv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf\" (UID: \"60c62d6a-e1ab-4d63-957b-346b65ec2263\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"
Sep 15 12:30:55.886687 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:55.886654    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"
Sep 15 12:30:56.002595 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:56.002564    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:30:56.163633 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:56.163611    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"]
Sep 15 12:30:56.165669 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:30:56.165640    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c62d6a_e1ab_4d63_957b_346b65ec2263.slice/crio-384b18bc0ae7e113ae3fb73e6a14154593439a59528131be77d3ed5d90fc586b WatchSource:0}: Error finding container 384b18bc0ae7e113ae3fb73e6a14154593439a59528131be77d3ed5d90fc586b: Status 404 returned error can't find the container with id 384b18bc0ae7e113ae3fb73e6a14154593439a59528131be77d3ed5d90fc586b
Sep 15 12:30:56.391449 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:56.391411    2579 generic.go:356] "Generic (PLEG): container finished" podID="60c62d6a-e1ab-4d63-957b-346b65ec2263" containerID="f1b74fbdd739466290197dcca7196e4584134800b6222f310f093515322f3b6a" exitCode=0
Sep 15 12:30:56.391616 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:56.391494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf" event={"ID":"60c62d6a-e1ab-4d63-957b-346b65ec2263","Type":"ContainerDied","Data":"f1b74fbdd739466290197dcca7196e4584134800b6222f310f093515322f3b6a"}
Sep 15 12:30:56.391616 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:56.391545    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf" event={"ID":"60c62d6a-e1ab-4d63-957b-346b65ec2263","Type":"ContainerStarted","Data":"384b18bc0ae7e113ae3fb73e6a14154593439a59528131be77d3ed5d90fc586b"}
Sep 15 12:30:57.397404 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:57.397372    2579 generic.go:356] "Generic (PLEG): container finished" podID="60c62d6a-e1ab-4d63-957b-346b65ec2263" containerID="f50009ed422cfbded9659b9591860941f3ab9da241d1431d21c77e65d07623fe" exitCode=0
Sep 15 12:30:57.397782 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:57.397445    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf" event={"ID":"60c62d6a-e1ab-4d63-957b-346b65ec2263","Type":"ContainerDied","Data":"f50009ed422cfbded9659b9591860941f3ab9da241d1431d21c77e65d07623fe"}
Sep 15 12:30:58.402978 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:58.402940    2579 generic.go:356] "Generic (PLEG): container finished" podID="60c62d6a-e1ab-4d63-957b-346b65ec2263" containerID="8b12d6c63e07239858db437d56b840923875a3750c0995696e3a03c9de7372b9" exitCode=0
Sep 15 12:30:58.403481 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:58.402998    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf" event={"ID":"60c62d6a-e1ab-4d63-957b-346b65ec2263","Type":"ContainerDied","Data":"8b12d6c63e07239858db437d56b840923875a3750c0995696e3a03c9de7372b9"}
Sep 15 12:30:59.526430 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:59.526407    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"
Sep 15 12:30:59.653944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:59.653913    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60c62d6a-e1ab-4d63-957b-346b65ec2263-util\") pod \"60c62d6a-e1ab-4d63-957b-346b65ec2263\" (UID: \"60c62d6a-e1ab-4d63-957b-346b65ec2263\") "
Sep 15 12:30:59.654108 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:59.653958    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z77rv\" (UniqueName: \"kubernetes.io/projected/60c62d6a-e1ab-4d63-957b-346b65ec2263-kube-api-access-z77rv\") pod \"60c62d6a-e1ab-4d63-957b-346b65ec2263\" (UID: \"60c62d6a-e1ab-4d63-957b-346b65ec2263\") "
Sep 15 12:30:59.654108 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:59.654030    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60c62d6a-e1ab-4d63-957b-346b65ec2263-bundle\") pod \"60c62d6a-e1ab-4d63-957b-346b65ec2263\" (UID: \"60c62d6a-e1ab-4d63-957b-346b65ec2263\") "
Sep 15 12:30:59.654747 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:59.654713    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c62d6a-e1ab-4d63-957b-346b65ec2263-bundle" (OuterVolumeSpecName: "bundle") pod "60c62d6a-e1ab-4d63-957b-346b65ec2263" (UID: "60c62d6a-e1ab-4d63-957b-346b65ec2263"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:30:59.656017 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:59.655993    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c62d6a-e1ab-4d63-957b-346b65ec2263-kube-api-access-z77rv" (OuterVolumeSpecName: "kube-api-access-z77rv") pod "60c62d6a-e1ab-4d63-957b-346b65ec2263" (UID: "60c62d6a-e1ab-4d63-957b-346b65ec2263"). InnerVolumeSpecName "kube-api-access-z77rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:30:59.662295 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:59.662264    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c62d6a-e1ab-4d63-957b-346b65ec2263-util" (OuterVolumeSpecName: "util") pod "60c62d6a-e1ab-4d63-957b-346b65ec2263" (UID: "60c62d6a-e1ab-4d63-957b-346b65ec2263"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:30:59.754580 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:59.754544    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z77rv\" (UniqueName: \"kubernetes.io/projected/60c62d6a-e1ab-4d63-957b-346b65ec2263-kube-api-access-z77rv\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:30:59.754580 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:59.754576    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60c62d6a-e1ab-4d63-957b-346b65ec2263-bundle\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:30:59.754580 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:30:59.754587    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60c62d6a-e1ab-4d63-957b-346b65ec2263-util\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:00.222913 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.222869    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:00.266010 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.265985    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"]
Sep 15 12:31:00.266252 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.266242    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c62d6a-e1ab-4d63-957b-346b65ec2263" containerName="pull"
Sep 15 12:31:00.266294 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.266255    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c62d6a-e1ab-4d63-957b-346b65ec2263" containerName="pull"
Sep 15 12:31:00.266294 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.266268    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c62d6a-e1ab-4d63-957b-346b65ec2263" containerName="util"
Sep 15 12:31:00.266294 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.266273    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c62d6a-e1ab-4d63-957b-346b65ec2263" containerName="util"
Sep 15 12:31:00.266294 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.266286    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c62d6a-e1ab-4d63-957b-346b65ec2263" containerName="extract"
Sep 15 12:31:00.266294 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.266292    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c62d6a-e1ab-4d63-957b-346b65ec2263" containerName="extract"
Sep 15 12:31:00.266446 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.266337    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c62d6a-e1ab-4d63-957b-346b65ec2263" containerName="extract"
Sep 15 12:31:00.269465 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.269443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:00.273146 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.273113    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5mcvl\""
Sep 15 12:31:00.273146 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.273125    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 12:31:00.273146 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.273122    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 12:31:00.273342 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.273172    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 12:31:00.277781 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.277762    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"]
Sep 15 12:31:00.321139 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.321113    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:00.360186 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.360161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m4wk\" (UniqueName: \"kubernetes.io/projected/34d45520-8fca-4d9b-9e36-b6b16b727e8c-kube-api-access-6m4wk\") pod \"lws-controller-manager-b47d9b998-9vlr6\" (UID: \"34d45520-8fca-4d9b-9e36-b6b16b727e8c\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:00.360306 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.360200    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/34d45520-8fca-4d9b-9e36-b6b16b727e8c-metrics-cert\") pod \"lws-controller-manager-b47d9b998-9vlr6\" (UID: \"34d45520-8fca-4d9b-9e36-b6b16b727e8c\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:00.360306 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.360227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/34d45520-8fca-4d9b-9e36-b6b16b727e8c-manager-config\") pod \"lws-controller-manager-b47d9b998-9vlr6\" (UID: \"34d45520-8fca-4d9b-9e36-b6b16b727e8c\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:00.360384 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.360332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34d45520-8fca-4d9b-9e36-b6b16b727e8c-cert\") pod \"lws-controller-manager-b47d9b998-9vlr6\" (UID: \"34d45520-8fca-4d9b-9e36-b6b16b727e8c\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:00.413029 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.412999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf" event={"ID":"60c62d6a-e1ab-4d63-957b-346b65ec2263","Type":"ContainerDied","Data":"384b18bc0ae7e113ae3fb73e6a14154593439a59528131be77d3ed5d90fc586b"}
Sep 15 12:31:00.413029 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.413026    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktmpf"
Sep 15 12:31:00.413188 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.413042    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="384b18bc0ae7e113ae3fb73e6a14154593439a59528131be77d3ed5d90fc586b"
Sep 15 12:31:00.461084 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.461054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/34d45520-8fca-4d9b-9e36-b6b16b727e8c-metrics-cert\") pod \"lws-controller-manager-b47d9b998-9vlr6\" (UID: \"34d45520-8fca-4d9b-9e36-b6b16b727e8c\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:00.461203 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.461096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/34d45520-8fca-4d9b-9e36-b6b16b727e8c-manager-config\") pod \"lws-controller-manager-b47d9b998-9vlr6\" (UID: \"34d45520-8fca-4d9b-9e36-b6b16b727e8c\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:00.461203 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.461143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34d45520-8fca-4d9b-9e36-b6b16b727e8c-cert\") pod \"lws-controller-manager-b47d9b998-9vlr6\" (UID: \"34d45520-8fca-4d9b-9e36-b6b16b727e8c\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:00.461203 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.461164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4wk\" (UniqueName: \"kubernetes.io/projected/34d45520-8fca-4d9b-9e36-b6b16b727e8c-kube-api-access-6m4wk\") pod \"lws-controller-manager-b47d9b998-9vlr6\" (UID: \"34d45520-8fca-4d9b-9e36-b6b16b727e8c\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:00.461861 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.461823    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/34d45520-8fca-4d9b-9e36-b6b16b727e8c-manager-config\") pod \"lws-controller-manager-b47d9b998-9vlr6\" (UID: \"34d45520-8fca-4d9b-9e36-b6b16b727e8c\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:00.463575 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.463554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/34d45520-8fca-4d9b-9e36-b6b16b727e8c-metrics-cert\") pod \"lws-controller-manager-b47d9b998-9vlr6\" (UID: \"34d45520-8fca-4d9b-9e36-b6b16b727e8c\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:00.463672 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.463649    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34d45520-8fca-4d9b-9e36-b6b16b727e8c-cert\") pod \"lws-controller-manager-b47d9b998-9vlr6\" (UID: \"34d45520-8fca-4d9b-9e36-b6b16b727e8c\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:00.469627 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.469597    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m4wk\" (UniqueName: \"kubernetes.io/projected/34d45520-8fca-4d9b-9e36-b6b16b727e8c-kube-api-access-6m4wk\") pod \"lws-controller-manager-b47d9b998-9vlr6\" (UID: \"34d45520-8fca-4d9b-9e36-b6b16b727e8c\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:00.580049 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.579961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:00.865168 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:00.865144    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"]
Sep 15 12:31:00.867648 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:31:00.867624    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d45520_8fca_4d9b_9e36_b6b16b727e8c.slice/crio-0107219fee265c2ea22654329d8e445a0da5867b1896e06d5d104a51dde9d374 WatchSource:0}: Error finding container 0107219fee265c2ea22654329d8e445a0da5867b1896e06d5d104a51dde9d374: Status 404 returned error can't find the container with id 0107219fee265c2ea22654329d8e445a0da5867b1896e06d5d104a51dde9d374
Sep 15 12:31:01.420798 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:01.420760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6" event={"ID":"34d45520-8fca-4d9b-9e36-b6b16b727e8c","Type":"ContainerStarted","Data":"0107219fee265c2ea22654329d8e445a0da5867b1896e06d5d104a51dde9d374"}
Sep 15 12:31:03.431731 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:03.431689    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6" event={"ID":"34d45520-8fca-4d9b-9e36-b6b16b727e8c","Type":"ContainerStarted","Data":"8c20a7b49c5c88a298dd8e9b8be20eb84fd788471406cfd45c1593cbfefab4f7"}
Sep 15 12:31:03.432145 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:03.431823    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:03.447616 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:03.447556    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6" podStartSLOduration=1.7410692939999999 podStartE2EDuration="3.447535745s" totalImagesPullingTime="1.706466451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:31:03.447174702 +0000 UTC m=+334.831885621" watchObservedRunningTime="2026-09-15 12:31:03.447535745 +0000 UTC m=+334.832246629"
Sep 15 12:31:07.054222 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:07.054158    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:08.331361 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:08.331315    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:09.720725 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.720689    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"]
Sep 15 12:31:09.724103 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.724084    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.726970 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.726942    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 12:31:09.727292 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.727275    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 12:31:09.727393 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.727374    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 12:31:09.727610 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.727598    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k9sn2\""
Sep 15 12:31:09.727922 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.727902    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 12:31:09.734452 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.734432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bdb03845-14e9-4cf4-a957-6fc95cade506-apiservice-cert\") pod \"opendatahub-operator-controller-manager-665cd58c4f-hb9xb\" (UID: \"bdb03845-14e9-4cf4-a957-6fc95cade506\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.734582 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.734466    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdb03845-14e9-4cf4-a957-6fc95cade506-tmp\") pod \"opendatahub-operator-controller-manager-665cd58c4f-hb9xb\" (UID: \"bdb03845-14e9-4cf4-a957-6fc95cade506\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.734582 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.734556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bdb03845-14e9-4cf4-a957-6fc95cade506-webhook-cert\") pod \"opendatahub-operator-controller-manager-665cd58c4f-hb9xb\" (UID: \"bdb03845-14e9-4cf4-a957-6fc95cade506\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.734667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.734580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bdb03845-14e9-4cf4-a957-6fc95cade506-manifests\") pod \"opendatahub-operator-controller-manager-665cd58c4f-hb9xb\" (UID: \"bdb03845-14e9-4cf4-a957-6fc95cade506\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.734667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.734606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4272\" (UniqueName: \"kubernetes.io/projected/bdb03845-14e9-4cf4-a957-6fc95cade506-kube-api-access-n4272\") pod \"opendatahub-operator-controller-manager-665cd58c4f-hb9xb\" (UID: \"bdb03845-14e9-4cf4-a957-6fc95cade506\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.746185 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.746161    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"]
Sep 15 12:31:09.835648 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.835612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bdb03845-14e9-4cf4-a957-6fc95cade506-webhook-cert\") pod \"opendatahub-operator-controller-manager-665cd58c4f-hb9xb\" (UID: \"bdb03845-14e9-4cf4-a957-6fc95cade506\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.835648 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.835648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bdb03845-14e9-4cf4-a957-6fc95cade506-manifests\") pod \"opendatahub-operator-controller-manager-665cd58c4f-hb9xb\" (UID: \"bdb03845-14e9-4cf4-a957-6fc95cade506\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.835929 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.835680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4272\" (UniqueName: \"kubernetes.io/projected/bdb03845-14e9-4cf4-a957-6fc95cade506-kube-api-access-n4272\") pod \"opendatahub-operator-controller-manager-665cd58c4f-hb9xb\" (UID: \"bdb03845-14e9-4cf4-a957-6fc95cade506\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.835929 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.835709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bdb03845-14e9-4cf4-a957-6fc95cade506-apiservice-cert\") pod \"opendatahub-operator-controller-manager-665cd58c4f-hb9xb\" (UID: \"bdb03845-14e9-4cf4-a957-6fc95cade506\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.835929 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.835748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdb03845-14e9-4cf4-a957-6fc95cade506-tmp\") pod \"opendatahub-operator-controller-manager-665cd58c4f-hb9xb\" (UID: \"bdb03845-14e9-4cf4-a957-6fc95cade506\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.836128 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.836105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bdb03845-14e9-4cf4-a957-6fc95cade506-manifests\") pod \"opendatahub-operator-controller-manager-665cd58c4f-hb9xb\" (UID: \"bdb03845-14e9-4cf4-a957-6fc95cade506\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.838092 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.838056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdb03845-14e9-4cf4-a957-6fc95cade506-tmp\") pod \"opendatahub-operator-controller-manager-665cd58c4f-hb9xb\" (UID: \"bdb03845-14e9-4cf4-a957-6fc95cade506\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.838239 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.838216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bdb03845-14e9-4cf4-a957-6fc95cade506-apiservice-cert\") pod \"opendatahub-operator-controller-manager-665cd58c4f-hb9xb\" (UID: \"bdb03845-14e9-4cf4-a957-6fc95cade506\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.838307 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.838255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bdb03845-14e9-4cf4-a957-6fc95cade506-webhook-cert\") pod \"opendatahub-operator-controller-manager-665cd58c4f-hb9xb\" (UID: \"bdb03845-14e9-4cf4-a957-6fc95cade506\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.845985 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.845963    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4272\" (UniqueName: \"kubernetes.io/projected/bdb03845-14e9-4cf4-a957-6fc95cade506-kube-api-access-n4272\") pod \"opendatahub-operator-controller-manager-665cd58c4f-hb9xb\" (UID: \"bdb03845-14e9-4cf4-a957-6fc95cade506\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:09.874377 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.874345    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"]
Sep 15 12:31:09.877778 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.877763    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"
Sep 15 12:31:09.880366 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.880345    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 12:31:09.880478 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.880457    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 12:31:09.880521 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.880469    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mtgf6\""
Sep 15 12:31:09.887384 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.887362    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"]
Sep 15 12:31:09.936770 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.936737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptvlz\" (UniqueName: \"kubernetes.io/projected/c6bddaf8-398b-4551-8872-6ec1b8be8366-kube-api-access-ptvlz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv\" (UID: \"c6bddaf8-398b-4551-8872-6ec1b8be8366\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"
Sep 15 12:31:09.936770 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.936770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6bddaf8-398b-4551-8872-6ec1b8be8366-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv\" (UID: \"c6bddaf8-398b-4551-8872-6ec1b8be8366\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"
Sep 15 12:31:09.936979 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:09.936810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6bddaf8-398b-4551-8872-6ec1b8be8366-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv\" (UID: \"c6bddaf8-398b-4551-8872-6ec1b8be8366\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"
Sep 15 12:31:10.034359 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:10.034262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:10.037556 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:10.037135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6bddaf8-398b-4551-8872-6ec1b8be8366-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv\" (UID: \"c6bddaf8-398b-4551-8872-6ec1b8be8366\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"
Sep 15 12:31:10.037556 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:10.037193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptvlz\" (UniqueName: \"kubernetes.io/projected/c6bddaf8-398b-4551-8872-6ec1b8be8366-kube-api-access-ptvlz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv\" (UID: \"c6bddaf8-398b-4551-8872-6ec1b8be8366\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"
Sep 15 12:31:10.037556 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:10.037215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6bddaf8-398b-4551-8872-6ec1b8be8366-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv\" (UID: \"c6bddaf8-398b-4551-8872-6ec1b8be8366\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"
Sep 15 12:31:10.037556 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:10.037550    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6bddaf8-398b-4551-8872-6ec1b8be8366-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv\" (UID: \"c6bddaf8-398b-4551-8872-6ec1b8be8366\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"
Sep 15 12:31:10.037768 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:10.037565    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6bddaf8-398b-4551-8872-6ec1b8be8366-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv\" (UID: \"c6bddaf8-398b-4551-8872-6ec1b8be8366\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"
Sep 15 12:31:10.046411 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:10.046385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptvlz\" (UniqueName: \"kubernetes.io/projected/c6bddaf8-398b-4551-8872-6ec1b8be8366-kube-api-access-ptvlz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv\" (UID: \"c6bddaf8-398b-4551-8872-6ec1b8be8366\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"
Sep 15 12:31:10.187710 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:10.187674    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"
Sep 15 12:31:10.315232 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:10.315160    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"]
Sep 15 12:31:10.318890 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:31:10.318862    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb03845_14e9_4cf4_a957_6fc95cade506.slice/crio-590917dd82bacdd997d4fc29208575691991072ef19af6734213e3054a0f09ec WatchSource:0}: Error finding container 590917dd82bacdd997d4fc29208575691991072ef19af6734213e3054a0f09ec: Status 404 returned error can't find the container with id 590917dd82bacdd997d4fc29208575691991072ef19af6734213e3054a0f09ec
Sep 15 12:31:10.460510 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:10.460483    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"]
Sep 15 12:31:10.461012 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:31:10.460988    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6bddaf8_398b_4551_8872_6ec1b8be8366.slice/crio-503bc9b850f09e71661fa8b8bcfd1992ea8fbcc83be458d0a0ac1b1a494c6ca6 WatchSource:0}: Error finding container 503bc9b850f09e71661fa8b8bcfd1992ea8fbcc83be458d0a0ac1b1a494c6ca6: Status 404 returned error can't find the container with id 503bc9b850f09e71661fa8b8bcfd1992ea8fbcc83be458d0a0ac1b1a494c6ca6
Sep 15 12:31:10.462541 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:10.462514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb" event={"ID":"bdb03845-14e9-4cf4-a957-6fc95cade506","Type":"ContainerStarted","Data":"590917dd82bacdd997d4fc29208575691991072ef19af6734213e3054a0f09ec"}
Sep 15 12:31:11.469988 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:11.469944    2579 generic.go:356] "Generic (PLEG): container finished" podID="c6bddaf8-398b-4551-8872-6ec1b8be8366" containerID="87c888a9e2c6969d416d77062bdbe30563969e16715f92488cc7d1c2e57d0a55" exitCode=0
Sep 15 12:31:11.470463 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:11.470041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv" event={"ID":"c6bddaf8-398b-4551-8872-6ec1b8be8366","Type":"ContainerDied","Data":"87c888a9e2c6969d416d77062bdbe30563969e16715f92488cc7d1c2e57d0a55"}
Sep 15 12:31:11.470463 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:11.470086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv" event={"ID":"c6bddaf8-398b-4551-8872-6ec1b8be8366","Type":"ContainerStarted","Data":"503bc9b850f09e71661fa8b8bcfd1992ea8fbcc83be458d0a0ac1b1a494c6ca6"}
Sep 15 12:31:11.599628 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:11.599596    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:11.697236 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:11.697195    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:12.477665 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:12.477627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv" event={"ID":"c6bddaf8-398b-4551-8872-6ec1b8be8366","Type":"ContainerStarted","Data":"fb0f5703bd3a6168c8c15aee40848278d9cecd12d0a2bea5cc8fe820664c12ad"}
Sep 15 12:31:14.437959 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:14.437917    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-b47d9b998-9vlr6"
Sep 15 12:31:14.491234 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:14.491199    2579 generic.go:356] "Generic (PLEG): container finished" podID="c6bddaf8-398b-4551-8872-6ec1b8be8366" containerID="fb0f5703bd3a6168c8c15aee40848278d9cecd12d0a2bea5cc8fe820664c12ad" exitCode=0
Sep 15 12:31:14.491413 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:14.491238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv" event={"ID":"c6bddaf8-398b-4551-8872-6ec1b8be8366","Type":"ContainerDied","Data":"fb0f5703bd3a6168c8c15aee40848278d9cecd12d0a2bea5cc8fe820664c12ad"}
Sep 15 12:31:14.493337 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:14.493305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb" event={"ID":"bdb03845-14e9-4cf4-a957-6fc95cade506","Type":"ContainerStarted","Data":"5efb39e92b9103c44bef3abe60010daae05b53aaa11b9b30ec9387d737b84d32"}
Sep 15 12:31:15.499587 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:15.499558    2579 generic.go:356] "Generic (PLEG): container finished" podID="c6bddaf8-398b-4551-8872-6ec1b8be8366" containerID="82b3cf78f3527e18098bc76f9a691fcc41f19739c6ac8aaf4b55b87f8a6c5a92" exitCode=0
Sep 15 12:31:15.500042 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:15.499629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv" event={"ID":"c6bddaf8-398b-4551-8872-6ec1b8be8366","Type":"ContainerDied","Data":"82b3cf78f3527e18098bc76f9a691fcc41f19739c6ac8aaf4b55b87f8a6c5a92"}
Sep 15 12:31:15.501169 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:15.501150    2579 generic.go:356] "Generic (PLEG): container finished" podID="bdb03845-14e9-4cf4-a957-6fc95cade506" containerID="5efb39e92b9103c44bef3abe60010daae05b53aaa11b9b30ec9387d737b84d32" exitCode=0
Sep 15 12:31:15.501281 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:15.501228    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb" event={"ID":"bdb03845-14e9-4cf4-a957-6fc95cade506","Type":"ContainerDied","Data":"5efb39e92b9103c44bef3abe60010daae05b53aaa11b9b30ec9387d737b84d32"}
Sep 15 12:31:16.509078 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:16.509041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb" event={"ID":"bdb03845-14e9-4cf4-a957-6fc95cade506","Type":"ContainerStarted","Data":"04f25bfeca26c3ee064b7ead46bd23a2e71112c2ef46fb7aacd4a3844333588c"}
Sep 15 12:31:16.509491 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:16.509456    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:16.531216 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:16.531163    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb" podStartSLOduration=2.448893792 podStartE2EDuration="7.531147645s" totalImagesPullingTime="3.946340783s" totalInitContainerRuntime="1.13591307s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:31:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:31:16.52880883 +0000 UTC m=+347.913519725" watchObservedRunningTime="2026-09-15 12:31:16.531147645 +0000 UTC m=+347.915858529"
Sep 15 12:31:16.642584 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:16.642555    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"
Sep 15 12:31:16.695375 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:16.695338    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6bddaf8-398b-4551-8872-6ec1b8be8366-bundle\") pod \"c6bddaf8-398b-4551-8872-6ec1b8be8366\" (UID: \"c6bddaf8-398b-4551-8872-6ec1b8be8366\") "
Sep 15 12:31:16.695554 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:16.695401    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptvlz\" (UniqueName: \"kubernetes.io/projected/c6bddaf8-398b-4551-8872-6ec1b8be8366-kube-api-access-ptvlz\") pod \"c6bddaf8-398b-4551-8872-6ec1b8be8366\" (UID: \"c6bddaf8-398b-4551-8872-6ec1b8be8366\") "
Sep 15 12:31:16.695554 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:16.695453    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6bddaf8-398b-4551-8872-6ec1b8be8366-util\") pod \"c6bddaf8-398b-4551-8872-6ec1b8be8366\" (UID: \"c6bddaf8-398b-4551-8872-6ec1b8be8366\") "
Sep 15 12:31:16.696203 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:16.696174    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6bddaf8-398b-4551-8872-6ec1b8be8366-bundle" (OuterVolumeSpecName: "bundle") pod "c6bddaf8-398b-4551-8872-6ec1b8be8366" (UID: "c6bddaf8-398b-4551-8872-6ec1b8be8366"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:31:16.697604 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:16.697581    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6bddaf8-398b-4551-8872-6ec1b8be8366-kube-api-access-ptvlz" (OuterVolumeSpecName: "kube-api-access-ptvlz") pod "c6bddaf8-398b-4551-8872-6ec1b8be8366" (UID: "c6bddaf8-398b-4551-8872-6ec1b8be8366"). InnerVolumeSpecName "kube-api-access-ptvlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:31:16.701852 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:16.701805    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6bddaf8-398b-4551-8872-6ec1b8be8366-util" (OuterVolumeSpecName: "util") pod "c6bddaf8-398b-4551-8872-6ec1b8be8366" (UID: "c6bddaf8-398b-4551-8872-6ec1b8be8366"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:31:16.796806 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:16.796705    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6bddaf8-398b-4551-8872-6ec1b8be8366-bundle\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:16.796806 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:16.796732    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptvlz\" (UniqueName: \"kubernetes.io/projected/c6bddaf8-398b-4551-8872-6ec1b8be8366-kube-api-access-ptvlz\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:16.796806 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:16.796742    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6bddaf8-398b-4551-8872-6ec1b8be8366-util\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:31:17.515544 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:17.515465    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv"
Sep 15 12:31:17.515544 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:17.515487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0l27sv" event={"ID":"c6bddaf8-398b-4551-8872-6ec1b8be8366","Type":"ContainerDied","Data":"503bc9b850f09e71661fa8b8bcfd1992ea8fbcc83be458d0a0ac1b1a494c6ca6"}
Sep 15 12:31:17.515544 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:17.515537    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="503bc9b850f09e71661fa8b8bcfd1992ea8fbcc83be458d0a0ac1b1a494c6ca6"
Sep 15 12:31:28.521652 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:28.521576    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-hb9xb"
Sep 15 12:31:31.863121 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:31.863069    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:31.972302 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:31.972263    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:32.253713 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.253619    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:32.378603 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.378567    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx"]
Sep 15 12:31:32.379062 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.379042    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6bddaf8-398b-4551-8872-6ec1b8be8366" containerName="pull"
Sep 15 12:31:32.379169 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.379066    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6bddaf8-398b-4551-8872-6ec1b8be8366" containerName="pull"
Sep 15 12:31:32.379169 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.379085    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6bddaf8-398b-4551-8872-6ec1b8be8366" containerName="util"
Sep 15 12:31:32.379169 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.379094    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6bddaf8-398b-4551-8872-6ec1b8be8366" containerName="util"
Sep 15 12:31:32.379169 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.379114    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6bddaf8-398b-4551-8872-6ec1b8be8366" containerName="extract"
Sep 15 12:31:32.379169 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.379124    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6bddaf8-398b-4551-8872-6ec1b8be8366" containerName="extract"
Sep 15 12:31:32.379337 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.379200    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6bddaf8-398b-4551-8872-6ec1b8be8366" containerName="extract"
Sep 15 12:31:32.382738 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.382715    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx"
Sep 15 12:31:32.386691 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.386667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 12:31:32.386691 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.386688    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 12:31:32.386870 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.386734    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 12:31:32.386870 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.386807    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-km5v7\""
Sep 15 12:31:32.387040 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.387022    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 12:31:32.403154 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.403130    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx"]
Sep 15 12:31:32.460231 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.460197    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:32.521623 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.521538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm2mh\" (UniqueName: \"kubernetes.io/projected/42f652d8-12b5-4295-8281-5e27afe32abc-kube-api-access-nm2mh\") pod \"kube-auth-proxy-6956d8dd68-5w4fx\" (UID: \"42f652d8-12b5-4295-8281-5e27afe32abc\") " pod="openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx"
Sep 15 12:31:32.521623 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.521576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f652d8-12b5-4295-8281-5e27afe32abc-tmp\") pod \"kube-auth-proxy-6956d8dd68-5w4fx\" (UID: \"42f652d8-12b5-4295-8281-5e27afe32abc\") " pod="openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx"
Sep 15 12:31:32.521812 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.521644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f652d8-12b5-4295-8281-5e27afe32abc-tls-certs\") pod \"kube-auth-proxy-6956d8dd68-5w4fx\" (UID: \"42f652d8-12b5-4295-8281-5e27afe32abc\") " pod="openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx"
Sep 15 12:31:32.622732 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.622690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm2mh\" (UniqueName: \"kubernetes.io/projected/42f652d8-12b5-4295-8281-5e27afe32abc-kube-api-access-nm2mh\") pod \"kube-auth-proxy-6956d8dd68-5w4fx\" (UID: \"42f652d8-12b5-4295-8281-5e27afe32abc\") " pod="openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx"
Sep 15 12:31:32.622732 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.622731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f652d8-12b5-4295-8281-5e27afe32abc-tmp\") pod \"kube-auth-proxy-6956d8dd68-5w4fx\" (UID: \"42f652d8-12b5-4295-8281-5e27afe32abc\") " pod="openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx"
Sep 15 12:31:32.623009 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.622779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f652d8-12b5-4295-8281-5e27afe32abc-tls-certs\") pod \"kube-auth-proxy-6956d8dd68-5w4fx\" (UID: \"42f652d8-12b5-4295-8281-5e27afe32abc\") " pod="openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx"
Sep 15 12:31:32.625119 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.625093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f652d8-12b5-4295-8281-5e27afe32abc-tmp\") pod \"kube-auth-proxy-6956d8dd68-5w4fx\" (UID: \"42f652d8-12b5-4295-8281-5e27afe32abc\") " pod="openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx"
Sep 15 12:31:32.625304 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.625285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42f652d8-12b5-4295-8281-5e27afe32abc-tls-certs\") pod \"kube-auth-proxy-6956d8dd68-5w4fx\" (UID: \"42f652d8-12b5-4295-8281-5e27afe32abc\") " pod="openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx"
Sep 15 12:31:32.630250 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.630223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm2mh\" (UniqueName: \"kubernetes.io/projected/42f652d8-12b5-4295-8281-5e27afe32abc-kube-api-access-nm2mh\") pod \"kube-auth-proxy-6956d8dd68-5w4fx\" (UID: \"42f652d8-12b5-4295-8281-5e27afe32abc\") " pod="openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx"
Sep 15 12:31:32.693069 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.693038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx"
Sep 15 12:31:32.809158 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.809052    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:32.909057 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.909021    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:32.968266 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:32.968241    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx"]
Sep 15 12:31:32.969657 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:31:32.969631    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f652d8_12b5_4295_8281_5e27afe32abc.slice/crio-a09f6a21ebc7a342d9c4926ac5f3d83ce13ebacb4a0afb18bec9b35848f9c869 WatchSource:0}: Error finding container a09f6a21ebc7a342d9c4926ac5f3d83ce13ebacb4a0afb18bec9b35848f9c869: Status 404 returned error can't find the container with id a09f6a21ebc7a342d9c4926ac5f3d83ce13ebacb4a0afb18bec9b35848f9c869
Sep 15 12:31:33.582519 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:33.582464    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:33.588064 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:33.588027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx" event={"ID":"42f652d8-12b5-4295-8281-5e27afe32abc","Type":"ContainerStarted","Data":"a09f6a21ebc7a342d9c4926ac5f3d83ce13ebacb4a0afb18bec9b35848f9c869"}
Sep 15 12:31:33.646790 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:33.646741    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:34.542516 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:34.542410    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:36.603408 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:36.603377    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx" event={"ID":"42f652d8-12b5-4295-8281-5e27afe32abc","Type":"ContainerStarted","Data":"9f90e62d43244448f652d92bf16a46893e766f1b008e428889ee75ad8a7c5187"}
Sep 15 12:31:36.622061 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:36.622014    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6956d8dd68-5w4fx" podStartSLOduration=1.882380052 podStartE2EDuration="4.621999305s" totalImagesPullingTime="2.739619253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:31:36.621037681 +0000 UTC m=+368.005748577" watchObservedRunningTime="2026-09-15 12:31:36.621999305 +0000 UTC m=+368.006710189"
Sep 15 12:31:40.741536 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:40.741493    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:40.860188 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:40.860154    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:41.339894 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:41.339858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:41.571619 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:41.571586    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:42.261644 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:42.261612    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:45.504055 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:45.504012    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:45.554194 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:45.554160    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:51.831165 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:51.831120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:54.645988 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:54.645950    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:31:54.763183 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:31:54.763140    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:27.104486 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:27.104436    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:43.902221 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:43.902179    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:44.002461 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:44.002423    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:47.454721 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.454685    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"]
Sep 15 12:32:47.457071 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.457054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"
Sep 15 12:32:47.459424 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.459399    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 12:32:47.459535 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.459495    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 12:32:47.460587 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.460567    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mtgf6\""
Sep 15 12:32:47.465424 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.465401    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"]
Sep 15 12:32:47.540068 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.540035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvwt7\" (UniqueName: \"kubernetes.io/projected/805a4680-a545-4cc9-ae0e-105ec240e20c-kube-api-access-fvwt7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt\" (UID: \"805a4680-a545-4cc9-ae0e-105ec240e20c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"
Sep 15 12:32:47.540287 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.540086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/805a4680-a545-4cc9-ae0e-105ec240e20c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt\" (UID: \"805a4680-a545-4cc9-ae0e-105ec240e20c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"
Sep 15 12:32:47.540287 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.540186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/805a4680-a545-4cc9-ae0e-105ec240e20c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt\" (UID: \"805a4680-a545-4cc9-ae0e-105ec240e20c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"
Sep 15 12:32:47.553847 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.553811    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"]
Sep 15 12:32:47.556065 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.556040    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"
Sep 15 12:32:47.565359 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.565337    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"]
Sep 15 12:32:47.641341 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.641304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwt7\" (UniqueName: \"kubernetes.io/projected/805a4680-a545-4cc9-ae0e-105ec240e20c-kube-api-access-fvwt7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt\" (UID: \"805a4680-a545-4cc9-ae0e-105ec240e20c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"
Sep 15 12:32:47.641530 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.641356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/805a4680-a545-4cc9-ae0e-105ec240e20c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt\" (UID: \"805a4680-a545-4cc9-ae0e-105ec240e20c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"
Sep 15 12:32:47.641530 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.641392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/805a4680-a545-4cc9-ae0e-105ec240e20c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt\" (UID: \"805a4680-a545-4cc9-ae0e-105ec240e20c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"
Sep 15 12:32:47.641530 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.641415    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83393b34-a3a9-4bcb-9d98-8a5376a4be02-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2\" (UID: \"83393b34-a3a9-4bcb-9d98-8a5376a4be02\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"
Sep 15 12:32:47.641530 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.641432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sdqd\" (UniqueName: \"kubernetes.io/projected/83393b34-a3a9-4bcb-9d98-8a5376a4be02-kube-api-access-8sdqd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2\" (UID: \"83393b34-a3a9-4bcb-9d98-8a5376a4be02\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"
Sep 15 12:32:47.641530 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.641459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83393b34-a3a9-4bcb-9d98-8a5376a4be02-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2\" (UID: \"83393b34-a3a9-4bcb-9d98-8a5376a4be02\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"
Sep 15 12:32:47.641884 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.641824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/805a4680-a545-4cc9-ae0e-105ec240e20c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt\" (UID: \"805a4680-a545-4cc9-ae0e-105ec240e20c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"
Sep 15 12:32:47.641934 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.641871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/805a4680-a545-4cc9-ae0e-105ec240e20c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt\" (UID: \"805a4680-a545-4cc9-ae0e-105ec240e20c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"
Sep 15 12:32:47.649520 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.649495    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvwt7\" (UniqueName: \"kubernetes.io/projected/805a4680-a545-4cc9-ae0e-105ec240e20c-kube-api-access-fvwt7\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt\" (UID: \"805a4680-a545-4cc9-ae0e-105ec240e20c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"
Sep 15 12:32:47.652322 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.652295    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"]
Sep 15 12:32:47.654645 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.654631    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"
Sep 15 12:32:47.664217 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.664192    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"]
Sep 15 12:32:47.742693 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.742610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83393b34-a3a9-4bcb-9d98-8a5376a4be02-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2\" (UID: \"83393b34-a3a9-4bcb-9d98-8a5376a4be02\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"
Sep 15 12:32:47.742693 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.742659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efe06825-b00d-4aab-af76-df5cfbe4d0e3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp\" (UID: \"efe06825-b00d-4aab-af76-df5cfbe4d0e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"
Sep 15 12:32:47.742693 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.742686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-969mb\" (UniqueName: \"kubernetes.io/projected/efe06825-b00d-4aab-af76-df5cfbe4d0e3-kube-api-access-969mb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp\" (UID: \"efe06825-b00d-4aab-af76-df5cfbe4d0e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"
Sep 15 12:32:47.742970 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.742742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efe06825-b00d-4aab-af76-df5cfbe4d0e3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp\" (UID: \"efe06825-b00d-4aab-af76-df5cfbe4d0e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"
Sep 15 12:32:47.742970 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.742799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83393b34-a3a9-4bcb-9d98-8a5376a4be02-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2\" (UID: \"83393b34-a3a9-4bcb-9d98-8a5376a4be02\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"
Sep 15 12:32:47.742970 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.742820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sdqd\" (UniqueName: \"kubernetes.io/projected/83393b34-a3a9-4bcb-9d98-8a5376a4be02-kube-api-access-8sdqd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2\" (UID: \"83393b34-a3a9-4bcb-9d98-8a5376a4be02\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"
Sep 15 12:32:47.743125 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.743010    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83393b34-a3a9-4bcb-9d98-8a5376a4be02-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2\" (UID: \"83393b34-a3a9-4bcb-9d98-8a5376a4be02\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"
Sep 15 12:32:47.743180 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.743125    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83393b34-a3a9-4bcb-9d98-8a5376a4be02-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2\" (UID: \"83393b34-a3a9-4bcb-9d98-8a5376a4be02\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"
Sep 15 12:32:47.752376 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.752347    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"]
Sep 15 12:32:47.754581 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.754556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sdqd\" (UniqueName: \"kubernetes.io/projected/83393b34-a3a9-4bcb-9d98-8a5376a4be02-kube-api-access-8sdqd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2\" (UID: \"83393b34-a3a9-4bcb-9d98-8a5376a4be02\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"
Sep 15 12:32:47.754714 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.754700    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"
Sep 15 12:32:47.763228 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.763196    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"]
Sep 15 12:32:47.766809 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.766787    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"
Sep 15 12:32:47.843501 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.843461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d754bc3e-509c-4e15-98a4-d94428765d24-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr\" (UID: \"d754bc3e-509c-4e15-98a4-d94428765d24\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"
Sep 15 12:32:47.843659 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.843517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prrfh\" (UniqueName: \"kubernetes.io/projected/d754bc3e-509c-4e15-98a4-d94428765d24-kube-api-access-prrfh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr\" (UID: \"d754bc3e-509c-4e15-98a4-d94428765d24\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"
Sep 15 12:32:47.843659 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.843560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efe06825-b00d-4aab-af76-df5cfbe4d0e3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp\" (UID: \"efe06825-b00d-4aab-af76-df5cfbe4d0e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"
Sep 15 12:32:47.843659 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.843599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-969mb\" (UniqueName: \"kubernetes.io/projected/efe06825-b00d-4aab-af76-df5cfbe4d0e3-kube-api-access-969mb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp\" (UID: \"efe06825-b00d-4aab-af76-df5cfbe4d0e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"
Sep 15 12:32:47.844321 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.843687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efe06825-b00d-4aab-af76-df5cfbe4d0e3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp\" (UID: \"efe06825-b00d-4aab-af76-df5cfbe4d0e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"
Sep 15 12:32:47.844321 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.843736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d754bc3e-509c-4e15-98a4-d94428765d24-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr\" (UID: \"d754bc3e-509c-4e15-98a4-d94428765d24\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"
Sep 15 12:32:47.844321 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.844080    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efe06825-b00d-4aab-af76-df5cfbe4d0e3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp\" (UID: \"efe06825-b00d-4aab-af76-df5cfbe4d0e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"
Sep 15 12:32:47.844321 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.844103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efe06825-b00d-4aab-af76-df5cfbe4d0e3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp\" (UID: \"efe06825-b00d-4aab-af76-df5cfbe4d0e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"
Sep 15 12:32:47.852288 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.852264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-969mb\" (UniqueName: \"kubernetes.io/projected/efe06825-b00d-4aab-af76-df5cfbe4d0e3-kube-api-access-969mb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp\" (UID: \"efe06825-b00d-4aab-af76-df5cfbe4d0e3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"
Sep 15 12:32:47.866083 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.866059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"
Sep 15 12:32:47.945010 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.944978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d754bc3e-509c-4e15-98a4-d94428765d24-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr\" (UID: \"d754bc3e-509c-4e15-98a4-d94428765d24\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"
Sep 15 12:32:47.945010 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.945022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d754bc3e-509c-4e15-98a4-d94428765d24-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr\" (UID: \"d754bc3e-509c-4e15-98a4-d94428765d24\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"
Sep 15 12:32:47.945287 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.945044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prrfh\" (UniqueName: \"kubernetes.io/projected/d754bc3e-509c-4e15-98a4-d94428765d24-kube-api-access-prrfh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr\" (UID: \"d754bc3e-509c-4e15-98a4-d94428765d24\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"
Sep 15 12:32:47.945490 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.945464    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d754bc3e-509c-4e15-98a4-d94428765d24-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr\" (UID: \"d754bc3e-509c-4e15-98a4-d94428765d24\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"
Sep 15 12:32:47.945587 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.945560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d754bc3e-509c-4e15-98a4-d94428765d24-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr\" (UID: \"d754bc3e-509c-4e15-98a4-d94428765d24\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"
Sep 15 12:32:47.953694 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.953662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prrfh\" (UniqueName: \"kubernetes.io/projected/d754bc3e-509c-4e15-98a4-d94428765d24-kube-api-access-prrfh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr\" (UID: \"d754bc3e-509c-4e15-98a4-d94428765d24\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"
Sep 15 12:32:47.965521 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:47.965497    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"
Sep 15 12:32:48.042516 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.042479    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"]
Sep 15 12:32:48.043344 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:32:48.043317    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805a4680_a545_4cc9_ae0e_105ec240e20c.slice/crio-e6987d1290d7f8cc238ed754e89bba4c81655e781621b604120d5ad085446da2 WatchSource:0}: Error finding container e6987d1290d7f8cc238ed754e89bba4c81655e781621b604120d5ad085446da2: Status 404 returned error can't find the container with id e6987d1290d7f8cc238ed754e89bba4c81655e781621b604120d5ad085446da2
Sep 15 12:32:48.071788 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.071761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"
Sep 15 12:32:48.147818 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.146784    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"]
Sep 15 12:32:48.235401 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.235373    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"]
Sep 15 12:32:48.236684 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:32:48.236654    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe06825_b00d_4aab_af76_df5cfbe4d0e3.slice/crio-fa9f9725d3c5050afffdfae881dad57bc21e5aff6234127d435c2f04f6f0100c WatchSource:0}: Error finding container fa9f9725d3c5050afffdfae881dad57bc21e5aff6234127d435c2f04f6f0100c: Status 404 returned error can't find the container with id fa9f9725d3c5050afffdfae881dad57bc21e5aff6234127d435c2f04f6f0100c
Sep 15 12:32:48.346577 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.346541    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"]
Sep 15 12:32:48.350058 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:32:48.350033    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd754bc3e_509c_4e15_98a4_d94428765d24.slice/crio-4788a38ce9ab030998a73adf633d9b2244776baa9647a45395fcf2b45ddbf3c2 WatchSource:0}: Error finding container 4788a38ce9ab030998a73adf633d9b2244776baa9647a45395fcf2b45ddbf3c2: Status 404 returned error can't find the container with id 4788a38ce9ab030998a73adf633d9b2244776baa9647a45395fcf2b45ddbf3c2
Sep 15 12:32:48.933567 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.933533    2579 generic.go:356] "Generic (PLEG): container finished" podID="805a4680-a545-4cc9-ae0e-105ec240e20c" containerID="3aba6f0dc9318005bda8bb959563812594afb236822412134483f32a72fe046d" exitCode=0
Sep 15 12:32:48.934018 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.933625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt" event={"ID":"805a4680-a545-4cc9-ae0e-105ec240e20c","Type":"ContainerDied","Data":"3aba6f0dc9318005bda8bb959563812594afb236822412134483f32a72fe046d"}
Sep 15 12:32:48.934018 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.933683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt" event={"ID":"805a4680-a545-4cc9-ae0e-105ec240e20c","Type":"ContainerStarted","Data":"e6987d1290d7f8cc238ed754e89bba4c81655e781621b604120d5ad085446da2"}
Sep 15 12:32:48.935396 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.935372    2579 generic.go:356] "Generic (PLEG): container finished" podID="d754bc3e-509c-4e15-98a4-d94428765d24" containerID="4ba86dfd30b13390e93a5587e36ae62c236a54a60e3e62db59884737a29770ad" exitCode=0
Sep 15 12:32:48.935484 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.935455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr" event={"ID":"d754bc3e-509c-4e15-98a4-d94428765d24","Type":"ContainerDied","Data":"4ba86dfd30b13390e93a5587e36ae62c236a54a60e3e62db59884737a29770ad"}
Sep 15 12:32:48.935556 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.935485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr" event={"ID":"d754bc3e-509c-4e15-98a4-d94428765d24","Type":"ContainerStarted","Data":"4788a38ce9ab030998a73adf633d9b2244776baa9647a45395fcf2b45ddbf3c2"}
Sep 15 12:32:48.937024 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.937005    2579 generic.go:356] "Generic (PLEG): container finished" podID="efe06825-b00d-4aab-af76-df5cfbe4d0e3" containerID="cc672d2ccc3a4daff2bcae6372c71f6103b03e9a50518f1c08497039c5631cfa" exitCode=0
Sep 15 12:32:48.937122 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.937076    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp" event={"ID":"efe06825-b00d-4aab-af76-df5cfbe4d0e3","Type":"ContainerDied","Data":"cc672d2ccc3a4daff2bcae6372c71f6103b03e9a50518f1c08497039c5631cfa"}
Sep 15 12:32:48.937122 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.937106    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp" event={"ID":"efe06825-b00d-4aab-af76-df5cfbe4d0e3","Type":"ContainerStarted","Data":"fa9f9725d3c5050afffdfae881dad57bc21e5aff6234127d435c2f04f6f0100c"}
Sep 15 12:32:48.938869 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.938851    2579 generic.go:356] "Generic (PLEG): container finished" podID="83393b34-a3a9-4bcb-9d98-8a5376a4be02" containerID="2fbc3ecff87401d96aa65fcf41a7d1cc0f2c3f1dae3d2718b3545e4008a2c217" exitCode=0
Sep 15 12:32:48.938964 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.938893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2" event={"ID":"83393b34-a3a9-4bcb-9d98-8a5376a4be02","Type":"ContainerDied","Data":"2fbc3ecff87401d96aa65fcf41a7d1cc0f2c3f1dae3d2718b3545e4008a2c217"}
Sep 15 12:32:48.938964 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:48.938925    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2" event={"ID":"83393b34-a3a9-4bcb-9d98-8a5376a4be02","Type":"ContainerStarted","Data":"11b49cf91be11757cfe036f36e3b8e813dafef510d6341d6c25950e044407670"}
Sep 15 12:32:49.947228 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:49.947194    2579 generic.go:356] "Generic (PLEG): container finished" podID="805a4680-a545-4cc9-ae0e-105ec240e20c" containerID="1fe3028fd96f52fafa24f660eed03e91e78bbf3d31c175692c3230cf17892892" exitCode=0
Sep 15 12:32:49.947570 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:49.947274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt" event={"ID":"805a4680-a545-4cc9-ae0e-105ec240e20c","Type":"ContainerDied","Data":"1fe3028fd96f52fafa24f660eed03e91e78bbf3d31c175692c3230cf17892892"}
Sep 15 12:32:49.949353 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:49.949330    2579 generic.go:356] "Generic (PLEG): container finished" podID="d754bc3e-509c-4e15-98a4-d94428765d24" containerID="d4d844dc715fc6c87c3ee5652863c0ad69f717ad366d62b0878ceebeda826f02" exitCode=0
Sep 15 12:32:49.949441 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:49.949368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr" event={"ID":"d754bc3e-509c-4e15-98a4-d94428765d24","Type":"ContainerDied","Data":"d4d844dc715fc6c87c3ee5652863c0ad69f717ad366d62b0878ceebeda826f02"}
Sep 15 12:32:50.956559 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:50.956519    2579 generic.go:356] "Generic (PLEG): container finished" podID="805a4680-a545-4cc9-ae0e-105ec240e20c" containerID="c9a2db0b39c9c3eb5b050b82b852bb76bd210d45e1e543ecec759bef8137d820" exitCode=0
Sep 15 12:32:50.956975 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:50.956602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt" event={"ID":"805a4680-a545-4cc9-ae0e-105ec240e20c","Type":"ContainerDied","Data":"c9a2db0b39c9c3eb5b050b82b852bb76bd210d45e1e543ecec759bef8137d820"}
Sep 15 12:32:50.958715 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:50.958691    2579 generic.go:356] "Generic (PLEG): container finished" podID="d754bc3e-509c-4e15-98a4-d94428765d24" containerID="ae8a374dee1293b3fe5ff7d7c83a0afcc8a87fc2d5445c0230d6b973c3698419" exitCode=0
Sep 15 12:32:50.958826 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:50.958741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr" event={"ID":"d754bc3e-509c-4e15-98a4-d94428765d24","Type":"ContainerDied","Data":"ae8a374dee1293b3fe5ff7d7c83a0afcc8a87fc2d5445c0230d6b973c3698419"}
Sep 15 12:32:51.965285 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:51.965250    2579 generic.go:356] "Generic (PLEG): container finished" podID="efe06825-b00d-4aab-af76-df5cfbe4d0e3" containerID="46a5e3a371ed7ee50362adcd42112c925733e15b87877a135378aa4f50574da3" exitCode=0
Sep 15 12:32:51.965669 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:51.965334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp" event={"ID":"efe06825-b00d-4aab-af76-df5cfbe4d0e3","Type":"ContainerDied","Data":"46a5e3a371ed7ee50362adcd42112c925733e15b87877a135378aa4f50574da3"}
Sep 15 12:32:52.102265 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.102238    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"
Sep 15 12:32:52.132214 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.132179    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"
Sep 15 12:32:52.285483 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.285406    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/805a4680-a545-4cc9-ae0e-105ec240e20c-bundle\") pod \"805a4680-a545-4cc9-ae0e-105ec240e20c\" (UID: \"805a4680-a545-4cc9-ae0e-105ec240e20c\") "
Sep 15 12:32:52.285483 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.285470    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvwt7\" (UniqueName: \"kubernetes.io/projected/805a4680-a545-4cc9-ae0e-105ec240e20c-kube-api-access-fvwt7\") pod \"805a4680-a545-4cc9-ae0e-105ec240e20c\" (UID: \"805a4680-a545-4cc9-ae0e-105ec240e20c\") "
Sep 15 12:32:52.285681 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.285542    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/805a4680-a545-4cc9-ae0e-105ec240e20c-util\") pod \"805a4680-a545-4cc9-ae0e-105ec240e20c\" (UID: \"805a4680-a545-4cc9-ae0e-105ec240e20c\") "
Sep 15 12:32:52.285681 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.285581    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d754bc3e-509c-4e15-98a4-d94428765d24-bundle\") pod \"d754bc3e-509c-4e15-98a4-d94428765d24\" (UID: \"d754bc3e-509c-4e15-98a4-d94428765d24\") "
Sep 15 12:32:52.285681 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.285618    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prrfh\" (UniqueName: \"kubernetes.io/projected/d754bc3e-509c-4e15-98a4-d94428765d24-kube-api-access-prrfh\") pod \"d754bc3e-509c-4e15-98a4-d94428765d24\" (UID: \"d754bc3e-509c-4e15-98a4-d94428765d24\") "
Sep 15 12:32:52.285681 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.285640    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d754bc3e-509c-4e15-98a4-d94428765d24-util\") pod \"d754bc3e-509c-4e15-98a4-d94428765d24\" (UID: \"d754bc3e-509c-4e15-98a4-d94428765d24\") "
Sep 15 12:32:52.286160 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.286137    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d754bc3e-509c-4e15-98a4-d94428765d24-bundle" (OuterVolumeSpecName: "bundle") pod "d754bc3e-509c-4e15-98a4-d94428765d24" (UID: "d754bc3e-509c-4e15-98a4-d94428765d24"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:32:52.286228 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.286160    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805a4680-a545-4cc9-ae0e-105ec240e20c-bundle" (OuterVolumeSpecName: "bundle") pod "805a4680-a545-4cc9-ae0e-105ec240e20c" (UID: "805a4680-a545-4cc9-ae0e-105ec240e20c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:32:52.287688 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.287663    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d754bc3e-509c-4e15-98a4-d94428765d24-kube-api-access-prrfh" (OuterVolumeSpecName: "kube-api-access-prrfh") pod "d754bc3e-509c-4e15-98a4-d94428765d24" (UID: "d754bc3e-509c-4e15-98a4-d94428765d24"). InnerVolumeSpecName "kube-api-access-prrfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:32:52.287948 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.287928    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805a4680-a545-4cc9-ae0e-105ec240e20c-kube-api-access-fvwt7" (OuterVolumeSpecName: "kube-api-access-fvwt7") pod "805a4680-a545-4cc9-ae0e-105ec240e20c" (UID: "805a4680-a545-4cc9-ae0e-105ec240e20c"). InnerVolumeSpecName "kube-api-access-fvwt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:32:52.291770 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.291734    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805a4680-a545-4cc9-ae0e-105ec240e20c-util" (OuterVolumeSpecName: "util") pod "805a4680-a545-4cc9-ae0e-105ec240e20c" (UID: "805a4680-a545-4cc9-ae0e-105ec240e20c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:32:52.292207 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.292187    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d754bc3e-509c-4e15-98a4-d94428765d24-util" (OuterVolumeSpecName: "util") pod "d754bc3e-509c-4e15-98a4-d94428765d24" (UID: "d754bc3e-509c-4e15-98a4-d94428765d24"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:32:52.386756 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.386719    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/805a4680-a545-4cc9-ae0e-105ec240e20c-util\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:52.386756 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.386749    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d754bc3e-509c-4e15-98a4-d94428765d24-bundle\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:52.386756 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.386763    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prrfh\" (UniqueName: \"kubernetes.io/projected/d754bc3e-509c-4e15-98a4-d94428765d24-kube-api-access-prrfh\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:52.387003 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.386775    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d754bc3e-509c-4e15-98a4-d94428765d24-util\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:52.387003 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.386787    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/805a4680-a545-4cc9-ae0e-105ec240e20c-bundle\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:52.387003 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.386797    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvwt7\" (UniqueName: \"kubernetes.io/projected/805a4680-a545-4cc9-ae0e-105ec240e20c-kube-api-access-fvwt7\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:52.563040 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.562959    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:32:52.972450 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.972423    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr"
Sep 15 12:32:52.972900 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.972420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hm7xr" event={"ID":"d754bc3e-509c-4e15-98a4-d94428765d24","Type":"ContainerDied","Data":"4788a38ce9ab030998a73adf633d9b2244776baa9647a45395fcf2b45ddbf3c2"}
Sep 15 12:32:52.972900 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.972550    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4788a38ce9ab030998a73adf633d9b2244776baa9647a45395fcf2b45ddbf3c2"
Sep 15 12:32:52.974716 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.974694    2579 generic.go:356] "Generic (PLEG): container finished" podID="efe06825-b00d-4aab-af76-df5cfbe4d0e3" containerID="5658c9c74b024311a1dce65b94f49bdb0c579b3fa0672dfda4f8dc36629fd521" exitCode=0
Sep 15 12:32:52.974871 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.974795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp" event={"ID":"efe06825-b00d-4aab-af76-df5cfbe4d0e3","Type":"ContainerDied","Data":"5658c9c74b024311a1dce65b94f49bdb0c579b3fa0672dfda4f8dc36629fd521"}
Sep 15 12:32:52.976731 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.976712    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt" event={"ID":"805a4680-a545-4cc9-ae0e-105ec240e20c","Type":"ContainerDied","Data":"e6987d1290d7f8cc238ed754e89bba4c81655e781621b604120d5ad085446da2"}
Sep 15 12:32:52.976801 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.976741    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6987d1290d7f8cc238ed754e89bba4c81655e781621b604120d5ad085446da2"
Sep 15 12:32:52.976801 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:52.976752    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cnkhqt"
Sep 15 12:32:54.103932 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:54.103907    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"
Sep 15 12:32:54.202008 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:54.201978    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-969mb\" (UniqueName: \"kubernetes.io/projected/efe06825-b00d-4aab-af76-df5cfbe4d0e3-kube-api-access-969mb\") pod \"efe06825-b00d-4aab-af76-df5cfbe4d0e3\" (UID: \"efe06825-b00d-4aab-af76-df5cfbe4d0e3\") "
Sep 15 12:32:54.202215 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:54.202075    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efe06825-b00d-4aab-af76-df5cfbe4d0e3-bundle\") pod \"efe06825-b00d-4aab-af76-df5cfbe4d0e3\" (UID: \"efe06825-b00d-4aab-af76-df5cfbe4d0e3\") "
Sep 15 12:32:54.202215 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:54.202117    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efe06825-b00d-4aab-af76-df5cfbe4d0e3-util\") pod \"efe06825-b00d-4aab-af76-df5cfbe4d0e3\" (UID: \"efe06825-b00d-4aab-af76-df5cfbe4d0e3\") "
Sep 15 12:32:54.202754 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:54.202723    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe06825-b00d-4aab-af76-df5cfbe4d0e3-bundle" (OuterVolumeSpecName: "bundle") pod "efe06825-b00d-4aab-af76-df5cfbe4d0e3" (UID: "efe06825-b00d-4aab-af76-df5cfbe4d0e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:32:54.204550 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:54.204528    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe06825-b00d-4aab-af76-df5cfbe4d0e3-kube-api-access-969mb" (OuterVolumeSpecName: "kube-api-access-969mb") pod "efe06825-b00d-4aab-af76-df5cfbe4d0e3" (UID: "efe06825-b00d-4aab-af76-df5cfbe4d0e3"). InnerVolumeSpecName "kube-api-access-969mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:32:54.207037 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:54.207016    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe06825-b00d-4aab-af76-df5cfbe4d0e3-util" (OuterVolumeSpecName: "util") pod "efe06825-b00d-4aab-af76-df5cfbe4d0e3" (UID: "efe06825-b00d-4aab-af76-df5cfbe4d0e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:32:54.303096 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:54.303037    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efe06825-b00d-4aab-af76-df5cfbe4d0e3-bundle\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:54.303096 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:54.303056    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efe06825-b00d-4aab-af76-df5cfbe4d0e3-util\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:54.303096 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:54.303065    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-969mb\" (UniqueName: \"kubernetes.io/projected/efe06825-b00d-4aab-af76-df5cfbe4d0e3-kube-api-access-969mb\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:32:54.987805 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:54.987776    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp"
Sep 15 12:32:54.988063 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:54.987812    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avb7dp" event={"ID":"efe06825-b00d-4aab-af76-df5cfbe4d0e3","Type":"ContainerDied","Data":"fa9f9725d3c5050afffdfae881dad57bc21e5aff6234127d435c2f04f6f0100c"}
Sep 15 12:32:54.988063 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:54.987874    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa9f9725d3c5050afffdfae881dad57bc21e5aff6234127d435c2f04f6f0100c"
Sep 15 12:32:58.003309 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:58.003222    2579 generic.go:356] "Generic (PLEG): container finished" podID="83393b34-a3a9-4bcb-9d98-8a5376a4be02" containerID="6ea2c46a7064d49ca000eb4fb72ef72b0b57d1178b78df0fb0ab784e000c9f11" exitCode=0
Sep 15 12:32:58.003309 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:58.003300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2" event={"ID":"83393b34-a3a9-4bcb-9d98-8a5376a4be02","Type":"ContainerDied","Data":"6ea2c46a7064d49ca000eb4fb72ef72b0b57d1178b78df0fb0ab784e000c9f11"}
Sep 15 12:32:59.010619 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:59.010581    2579 generic.go:356] "Generic (PLEG): container finished" podID="83393b34-a3a9-4bcb-9d98-8a5376a4be02" containerID="19b0ad41bc9e868c31e6fb36ac69bbc3c4d4a743a98a7552463d46a80e944a6e" exitCode=0
Sep 15 12:32:59.011070 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:32:59.010671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2" event={"ID":"83393b34-a3a9-4bcb-9d98-8a5376a4be02","Type":"ContainerDied","Data":"19b0ad41bc9e868c31e6fb36ac69bbc3c4d4a743a98a7552463d46a80e944a6e"}
Sep 15 12:33:00.132294 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:00.132271    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"
Sep 15 12:33:00.153004 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:00.152976    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83393b34-a3a9-4bcb-9d98-8a5376a4be02-util\") pod \"83393b34-a3a9-4bcb-9d98-8a5376a4be02\" (UID: \"83393b34-a3a9-4bcb-9d98-8a5376a4be02\") "
Sep 15 12:33:00.153125 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:00.153013    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sdqd\" (UniqueName: \"kubernetes.io/projected/83393b34-a3a9-4bcb-9d98-8a5376a4be02-kube-api-access-8sdqd\") pod \"83393b34-a3a9-4bcb-9d98-8a5376a4be02\" (UID: \"83393b34-a3a9-4bcb-9d98-8a5376a4be02\") "
Sep 15 12:33:00.153125 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:00.153046    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83393b34-a3a9-4bcb-9d98-8a5376a4be02-bundle\") pod \"83393b34-a3a9-4bcb-9d98-8a5376a4be02\" (UID: \"83393b34-a3a9-4bcb-9d98-8a5376a4be02\") "
Sep 15 12:33:00.153583 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:00.153552    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83393b34-a3a9-4bcb-9d98-8a5376a4be02-bundle" (OuterVolumeSpecName: "bundle") pod "83393b34-a3a9-4bcb-9d98-8a5376a4be02" (UID: "83393b34-a3a9-4bcb-9d98-8a5376a4be02"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:33:00.155556 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:00.155528    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83393b34-a3a9-4bcb-9d98-8a5376a4be02-kube-api-access-8sdqd" (OuterVolumeSpecName: "kube-api-access-8sdqd") pod "83393b34-a3a9-4bcb-9d98-8a5376a4be02" (UID: "83393b34-a3a9-4bcb-9d98-8a5376a4be02"). InnerVolumeSpecName "kube-api-access-8sdqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:33:00.158114 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:00.158091    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83393b34-a3a9-4bcb-9d98-8a5376a4be02-util" (OuterVolumeSpecName: "util") pod "83393b34-a3a9-4bcb-9d98-8a5376a4be02" (UID: "83393b34-a3a9-4bcb-9d98-8a5376a4be02"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:33:00.254064 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:00.254024    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83393b34-a3a9-4bcb-9d98-8a5376a4be02-bundle\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:33:00.254064 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:00.254056    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83393b34-a3a9-4bcb-9d98-8a5376a4be02-util\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:33:00.254064 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:00.254065    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sdqd\" (UniqueName: \"kubernetes.io/projected/83393b34-a3a9-4bcb-9d98-8a5376a4be02-kube-api-access-8sdqd\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:33:01.021652 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:01.021621    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2" event={"ID":"83393b34-a3a9-4bcb-9d98-8a5376a4be02","Type":"ContainerDied","Data":"11b49cf91be11757cfe036f36e3b8e813dafef510d6341d6c25950e044407670"}
Sep 15 12:33:01.021825 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:01.021699    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11b49cf91be11757cfe036f36e3b8e813dafef510d6341d6c25950e044407670"
Sep 15 12:33:01.021825 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:01.021706    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cspn2"
Sep 15 12:33:09.431919 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:09.431880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:10.278071 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:10.278027    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:10.842253 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:10.842217    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:10.944419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:10.944386    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:13.476275 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:13.476238    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:13.670601 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:13.670562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:15.182850 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:15.182809    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:15.630233 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:15.630203    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:16.028673 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:16.028593    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:16.463143 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:16.463104    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:16.677348 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:16.677313    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:16.778082 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:16.777987    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:19.833273 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:19.833214    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:21.275280 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:21.275247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:22.834747 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:22.834709    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:22.934559 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:22.934529    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:23.231407 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:23.231366    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:23.331663 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:23.331615    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:26.453368 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.453331    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:26.459039 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459013    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv"]
Sep 15 12:33:26.459323 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459311    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805a4680-a545-4cc9-ae0e-105ec240e20c" containerName="util"
Sep 15 12:33:26.459364 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459326    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="805a4680-a545-4cc9-ae0e-105ec240e20c" containerName="util"
Sep 15 12:33:26.459364 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459334    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe06825-b00d-4aab-af76-df5cfbe4d0e3" containerName="extract"
Sep 15 12:33:26.459364 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459340    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe06825-b00d-4aab-af76-df5cfbe4d0e3" containerName="extract"
Sep 15 12:33:26.459364 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459351    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe06825-b00d-4aab-af76-df5cfbe4d0e3" containerName="util"
Sep 15 12:33:26.459364 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459356    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe06825-b00d-4aab-af76-df5cfbe4d0e3" containerName="util"
Sep 15 12:33:26.459364 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459364    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805a4680-a545-4cc9-ae0e-105ec240e20c" containerName="extract"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459370    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="805a4680-a545-4cc9-ae0e-105ec240e20c" containerName="extract"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459381    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d754bc3e-509c-4e15-98a4-d94428765d24" containerName="pull"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459388    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d754bc3e-509c-4e15-98a4-d94428765d24" containerName="pull"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459396    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d754bc3e-509c-4e15-98a4-d94428765d24" containerName="util"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459423    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d754bc3e-509c-4e15-98a4-d94428765d24" containerName="util"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459429    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe06825-b00d-4aab-af76-df5cfbe4d0e3" containerName="pull"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459435    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe06825-b00d-4aab-af76-df5cfbe4d0e3" containerName="pull"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459442    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83393b34-a3a9-4bcb-9d98-8a5376a4be02" containerName="util"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459447    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="83393b34-a3a9-4bcb-9d98-8a5376a4be02" containerName="util"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459456    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83393b34-a3a9-4bcb-9d98-8a5376a4be02" containerName="pull"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459464    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="83393b34-a3a9-4bcb-9d98-8a5376a4be02" containerName="pull"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459470    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83393b34-a3a9-4bcb-9d98-8a5376a4be02" containerName="extract"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459475    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="83393b34-a3a9-4bcb-9d98-8a5376a4be02" containerName="extract"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459482    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d754bc3e-509c-4e15-98a4-d94428765d24" containerName="extract"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459486    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d754bc3e-509c-4e15-98a4-d94428765d24" containerName="extract"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459493    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805a4680-a545-4cc9-ae0e-105ec240e20c" containerName="pull"
Sep 15 12:33:26.459538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459498    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="805a4680-a545-4cc9-ae0e-105ec240e20c" containerName="pull"
Sep 15 12:33:26.460300 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459559    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="805a4680-a545-4cc9-ae0e-105ec240e20c" containerName="extract"
Sep 15 12:33:26.460300 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459567    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="83393b34-a3a9-4bcb-9d98-8a5376a4be02" containerName="extract"
Sep 15 12:33:26.460300 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459573    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe06825-b00d-4aab-af76-df5cfbe4d0e3" containerName="extract"
Sep 15 12:33:26.460300 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.459581    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d754bc3e-509c-4e15-98a4-d94428765d24" containerName="extract"
Sep 15 12:33:26.462196 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.462180    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv"
Sep 15 12:33:26.464248 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.464227    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 15 12:33:26.464545 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.464512    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 15 12:33:26.464675 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.464632    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 12:33:26.464767 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.464747    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 12:33:26.465393 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.465371    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8mdnh\""
Sep 15 12:33:26.472774 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.472750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv"]
Sep 15 12:33:26.576342 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.576305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/91f8fa70-291c-4441-a53b-4260034f4fa8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nrbwv\" (UID: \"91f8fa70-291c-4441-a53b-4260034f4fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv"
Sep 15 12:33:26.576518 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.576360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/91f8fa70-291c-4441-a53b-4260034f4fa8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-nrbwv\" (UID: \"91f8fa70-291c-4441-a53b-4260034f4fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv"
Sep 15 12:33:26.576518 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.576394    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfvhs\" (UniqueName: \"kubernetes.io/projected/91f8fa70-291c-4441-a53b-4260034f4fa8-kube-api-access-kfvhs\") pod \"kuadrant-console-plugin-fbdd4f445-nrbwv\" (UID: \"91f8fa70-291c-4441-a53b-4260034f4fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv"
Sep 15 12:33:26.677166 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.677123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/91f8fa70-291c-4441-a53b-4260034f4fa8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nrbwv\" (UID: \"91f8fa70-291c-4441-a53b-4260034f4fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv"
Sep 15 12:33:26.677350 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.677187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/91f8fa70-291c-4441-a53b-4260034f4fa8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-nrbwv\" (UID: \"91f8fa70-291c-4441-a53b-4260034f4fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv"
Sep 15 12:33:26.677350 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.677229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfvhs\" (UniqueName: \"kubernetes.io/projected/91f8fa70-291c-4441-a53b-4260034f4fa8-kube-api-access-kfvhs\") pod \"kuadrant-console-plugin-fbdd4f445-nrbwv\" (UID: \"91f8fa70-291c-4441-a53b-4260034f4fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv"
Sep 15 12:33:26.677889 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.677866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/91f8fa70-291c-4441-a53b-4260034f4fa8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-nrbwv\" (UID: \"91f8fa70-291c-4441-a53b-4260034f4fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv"
Sep 15 12:33:26.679552 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.679532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/91f8fa70-291c-4441-a53b-4260034f4fa8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nrbwv\" (UID: \"91f8fa70-291c-4441-a53b-4260034f4fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv"
Sep 15 12:33:26.685439 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.685414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfvhs\" (UniqueName: \"kubernetes.io/projected/91f8fa70-291c-4441-a53b-4260034f4fa8-kube-api-access-kfvhs\") pod \"kuadrant-console-plugin-fbdd4f445-nrbwv\" (UID: \"91f8fa70-291c-4441-a53b-4260034f4fa8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv"
Sep 15 12:33:26.704529 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.704468    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:26.772506 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.772476    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv"
Sep 15 12:33:26.836860 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.836781    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:26.936575 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:26.936539    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:27.041180 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:27.041156    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv"]
Sep 15 12:33:27.043161 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:33:27.043136    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91f8fa70_291c_4441_a53b_4260034f4fa8.slice/crio-241f50756b53e5629c476be34e39a1d3eba22447043e6c54d60a41c45848473b WatchSource:0}: Error finding container 241f50756b53e5629c476be34e39a1d3eba22447043e6c54d60a41c45848473b: Status 404 returned error can't find the container with id 241f50756b53e5629c476be34e39a1d3eba22447043e6c54d60a41c45848473b
Sep 15 12:33:27.148692 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:27.148657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv" event={"ID":"91f8fa70-291c-4441-a53b-4260034f4fa8","Type":"ContainerStarted","Data":"241f50756b53e5629c476be34e39a1d3eba22447043e6c54d60a41c45848473b"}
Sep 15 12:33:32.176618 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:32.176579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv" event={"ID":"91f8fa70-291c-4441-a53b-4260034f4fa8","Type":"ContainerStarted","Data":"8417ebf7a071807a5cd128a4193d73cbbe589bcdc97021b186520f70cd591125"}
Sep 15 12:33:32.192349 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:32.192303    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nrbwv" podStartSLOduration=1.782481038 podStartE2EDuration="6.19228877s" totalImagesPullingTime="4.409807732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:33:32.19097812 +0000 UTC m=+483.575689006" watchObservedRunningTime="2026-09-15 12:33:32.19228877 +0000 UTC m=+483.576999654"
Sep 15 12:33:32.407902 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:32.407859    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:32.501720 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:32.501622    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:41.514054 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:41.514017    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:41.614622 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:41.614592    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:45.294964 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:45.294928    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:45.349534 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:45.349502    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:45.722956 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:45.722917    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:45.812235 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:45.812198    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:49.303526 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:49.303480    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:49.403867 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:49.403812    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:50.981766 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:50.981727    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:54.116007 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:54.115970    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:54.169202 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:54.169167    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:54.298129 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:54.298092    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:54.364885 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:54.364849    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:55.298649 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:55.298612    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:33:55.363992 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:33:55.363958    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:00.976230 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:00.976187    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:01.077659 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:01.077623    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:06.102747 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:06.102713    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:06.202504 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:06.202471    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:14.571346 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:14.571307    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:15.860688 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:15.860651    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:15.960765 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:15.960733    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:22.147086 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.147003    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:22.259161 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.259129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:22.316627 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.316584    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:22.548292 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.548214    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:34:22.566606 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.566576    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:34:22.566755 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.566688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-tfv49"
Sep 15 12:34:22.569321 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.569299    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 12:34:22.570317 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.570294    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:34:22.735079 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.735041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196-config-file\") pod \"limitador-limitador-84994f7ffc-tfv49\" (UID: \"431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tfv49"
Sep 15 12:34:22.735263 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.735114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmsnn\" (UniqueName: \"kubernetes.io/projected/431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196-kube-api-access-kmsnn\") pod \"limitador-limitador-84994f7ffc-tfv49\" (UID: \"431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tfv49"
Sep 15 12:34:22.835925 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.835819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196-config-file\") pod \"limitador-limitador-84994f7ffc-tfv49\" (UID: \"431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tfv49"
Sep 15 12:34:22.836093 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.835983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmsnn\" (UniqueName: \"kubernetes.io/projected/431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196-kube-api-access-kmsnn\") pod \"limitador-limitador-84994f7ffc-tfv49\" (UID: \"431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tfv49"
Sep 15 12:34:22.836499 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.836473    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196-config-file\") pod \"limitador-limitador-84994f7ffc-tfv49\" (UID: \"431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tfv49"
Sep 15 12:34:22.843919 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.843897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmsnn\" (UniqueName: \"kubernetes.io/projected/431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196-kube-api-access-kmsnn\") pod \"limitador-limitador-84994f7ffc-tfv49\" (UID: \"431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tfv49"
Sep 15 12:34:22.877374 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.877348    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-tfv49"
Sep 15 12:34:22.956077 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.956042    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-662hh"]
Sep 15 12:34:22.959458 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.959437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-662hh"
Sep 15 12:34:22.962163 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.962139    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jpchf\""
Sep 15 12:34:22.963559 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.963531    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-662hh"]
Sep 15 12:34:22.996276 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:22.996247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:23.039089 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:23.039062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dvcv\" (UniqueName: \"kubernetes.io/projected/613984c6-b796-4e56-b29b-c466fb3b3d82-kube-api-access-2dvcv\") pod \"authorino-7bcfd8f9cd-662hh\" (UID: \"613984c6-b796-4e56-b29b-c466fb3b3d82\") " pod="kuadrant-system/authorino-7bcfd8f9cd-662hh"
Sep 15 12:34:23.080785 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:23.080759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:23.140356 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:23.140327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dvcv\" (UniqueName: \"kubernetes.io/projected/613984c6-b796-4e56-b29b-c466fb3b3d82-kube-api-access-2dvcv\") pod \"authorino-7bcfd8f9cd-662hh\" (UID: \"613984c6-b796-4e56-b29b-c466fb3b3d82\") " pod="kuadrant-system/authorino-7bcfd8f9cd-662hh"
Sep 15 12:34:23.143917 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:23.143896    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:34:23.145922 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:34:23.145901    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431cdb7b_b8eb_4c2f_bdbd_4a0cb657b196.slice/crio-d4a162b548eed6fbe73ea9564decf1d07de74ce422c901a61ebf46d45bfd3d03 WatchSource:0}: Error finding container d4a162b548eed6fbe73ea9564decf1d07de74ce422c901a61ebf46d45bfd3d03: Status 404 returned error can't find the container with id d4a162b548eed6fbe73ea9564decf1d07de74ce422c901a61ebf46d45bfd3d03
Sep 15 12:34:23.148671 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:23.148650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dvcv\" (UniqueName: \"kubernetes.io/projected/613984c6-b796-4e56-b29b-c466fb3b3d82-kube-api-access-2dvcv\") pod \"authorino-7bcfd8f9cd-662hh\" (UID: \"613984c6-b796-4e56-b29b-c466fb3b3d82\") " pod="kuadrant-system/authorino-7bcfd8f9cd-662hh"
Sep 15 12:34:23.269965 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:23.269933    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-662hh"
Sep 15 12:34:23.417759 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:23.417674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-tfv49" event={"ID":"431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196","Type":"ContainerStarted","Data":"d4a162b548eed6fbe73ea9564decf1d07de74ce422c901a61ebf46d45bfd3d03"}
Sep 15 12:34:23.537994 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:23.537970    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-662hh"]
Sep 15 12:34:23.539611 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:34:23.539588    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod613984c6_b796_4e56_b29b_c466fb3b3d82.slice/crio-cb56ca2df89c67102f70ab4dbf8346326ebb821e729ca74dd08fdd78218e545e WatchSource:0}: Error finding container cb56ca2df89c67102f70ab4dbf8346326ebb821e729ca74dd08fdd78218e545e: Status 404 returned error can't find the container with id cb56ca2df89c67102f70ab4dbf8346326ebb821e729ca74dd08fdd78218e545e
Sep 15 12:34:24.426084 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:24.426045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-662hh" event={"ID":"613984c6-b796-4e56-b29b-c466fb3b3d82","Type":"ContainerStarted","Data":"cb56ca2df89c67102f70ab4dbf8346326ebb821e729ca74dd08fdd78218e545e"}
Sep 15 12:34:25.992650 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:25.992609    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-662hh"]
Sep 15 12:34:26.442677 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:26.442635    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-tfv49" event={"ID":"431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196","Type":"ContainerStarted","Data":"6b451b3fe1c2612bbaedeab6e6a9f6b68690dabfff721f675a4e2ac77b3f6b22"}
Sep 15 12:34:26.443111 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:26.442740    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-tfv49"
Sep 15 12:34:26.444108 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:26.444082    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-662hh" event={"ID":"613984c6-b796-4e56-b29b-c466fb3b3d82","Type":"ContainerStarted","Data":"f169b44b42de677ef3f387e3ba29a504888a615e1b328acfb81193b33af4a60a"}
Sep 15 12:34:26.444209 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:26.444191    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-662hh" podUID="613984c6-b796-4e56-b29b-c466fb3b3d82" containerName="authorino" containerID="cri-o://f169b44b42de677ef3f387e3ba29a504888a615e1b328acfb81193b33af4a60a" gracePeriod=30
Sep 15 12:34:26.459071 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:26.459018    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-tfv49" podStartSLOduration=1.485067701 podStartE2EDuration="4.45900351s" totalImagesPullingTime="2.973935809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:34:26.458620342 +0000 UTC m=+537.843331226" watchObservedRunningTime="2026-09-15 12:34:26.45900351 +0000 UTC m=+537.843714398"
Sep 15 12:34:26.472449 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:26.472407    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-662hh" podStartSLOduration=1.885098809 podStartE2EDuration="4.472398412s" totalImagesPullingTime="2.587299603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:34:26.470965363 +0000 UTC m=+537.855676247" watchObservedRunningTime="2026-09-15 12:34:26.472398412 +0000 UTC m=+537.857109296"
Sep 15 12:34:26.685087 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:26.685064    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-662hh"
Sep 15 12:34:26.773164 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:26.773085    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dvcv\" (UniqueName: \"kubernetes.io/projected/613984c6-b796-4e56-b29b-c466fb3b3d82-kube-api-access-2dvcv\") pod \"613984c6-b796-4e56-b29b-c466fb3b3d82\" (UID: \"613984c6-b796-4e56-b29b-c466fb3b3d82\") "
Sep 15 12:34:26.775232 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:26.775208    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613984c6-b796-4e56-b29b-c466fb3b3d82-kube-api-access-2dvcv" (OuterVolumeSpecName: "kube-api-access-2dvcv") pod "613984c6-b796-4e56-b29b-c466fb3b3d82" (UID: "613984c6-b796-4e56-b29b-c466fb3b3d82"). InnerVolumeSpecName "kube-api-access-2dvcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:34:26.874133 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:26.874087    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dvcv\" (UniqueName: \"kubernetes.io/projected/613984c6-b796-4e56-b29b-c466fb3b3d82-kube-api-access-2dvcv\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:34:27.450713 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:27.450616    2579 generic.go:356] "Generic (PLEG): container finished" podID="613984c6-b796-4e56-b29b-c466fb3b3d82" containerID="f169b44b42de677ef3f387e3ba29a504888a615e1b328acfb81193b33af4a60a" exitCode=0
Sep 15 12:34:27.450713 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:27.450656    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-662hh"
Sep 15 12:34:27.450713 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:27.450699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-662hh" event={"ID":"613984c6-b796-4e56-b29b-c466fb3b3d82","Type":"ContainerDied","Data":"f169b44b42de677ef3f387e3ba29a504888a615e1b328acfb81193b33af4a60a"}
Sep 15 12:34:27.451276 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:27.450737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-662hh" event={"ID":"613984c6-b796-4e56-b29b-c466fb3b3d82","Type":"ContainerDied","Data":"cb56ca2df89c67102f70ab4dbf8346326ebb821e729ca74dd08fdd78218e545e"}
Sep 15 12:34:27.451276 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:27.450753    2579 scope.go:117] "RemoveContainer" containerID="f169b44b42de677ef3f387e3ba29a504888a615e1b328acfb81193b33af4a60a"
Sep 15 12:34:27.459822 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:27.459805    2579 scope.go:117] "RemoveContainer" containerID="f169b44b42de677ef3f387e3ba29a504888a615e1b328acfb81193b33af4a60a"
Sep 15 12:34:27.460150 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:34:27.460133    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f169b44b42de677ef3f387e3ba29a504888a615e1b328acfb81193b33af4a60a\": container with ID starting with f169b44b42de677ef3f387e3ba29a504888a615e1b328acfb81193b33af4a60a not found: ID does not exist" containerID="f169b44b42de677ef3f387e3ba29a504888a615e1b328acfb81193b33af4a60a"
Sep 15 12:34:27.460196 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:27.460159    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f169b44b42de677ef3f387e3ba29a504888a615e1b328acfb81193b33af4a60a"} err="failed to get container status \"f169b44b42de677ef3f387e3ba29a504888a615e1b328acfb81193b33af4a60a\": rpc error: code = NotFound desc = could not find container \"f169b44b42de677ef3f387e3ba29a504888a615e1b328acfb81193b33af4a60a\": container with ID starting with f169b44b42de677ef3f387e3ba29a504888a615e1b328acfb81193b33af4a60a not found: ID does not exist"
Sep 15 12:34:27.468548 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:27.468526    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-662hh"]
Sep 15 12:34:27.470361 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:27.470343    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-662hh"]
Sep 15 12:34:29.105302 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:29.105276    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="613984c6-b796-4e56-b29b-c466fb3b3d82" path="/var/lib/kubelet/pods/613984c6-b796-4e56-b29b-c466fb3b3d82/volumes"
Sep 15 12:34:37.451706 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:37.451672    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-tfv49"
Sep 15 12:34:42.400133 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:42.400096    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:42.498246 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:42.498211    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:42.598543 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:42.598507    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:42.959414 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:42.959383    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:55.423096 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.423060    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:55.637955 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.637922    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-8zvx2"]
Sep 15 12:34:55.638299 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.638281    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="613984c6-b796-4e56-b29b-c466fb3b3d82" containerName="authorino"
Sep 15 12:34:55.638383 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.638303    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="613984c6-b796-4e56-b29b-c466fb3b3d82" containerName="authorino"
Sep 15 12:34:55.638383 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.638379    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="613984c6-b796-4e56-b29b-c466fb3b3d82" containerName="authorino"
Sep 15 12:34:55.641023 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.641004    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-8zvx2"
Sep 15 12:34:55.643406 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.643383    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jpchf\""
Sep 15 12:34:55.646444 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.646422    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-8zvx2"]
Sep 15 12:34:55.684170 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.684118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdpkt\" (UniqueName: \"kubernetes.io/projected/194cb818-d0b5-4c4d-b0de-293e7d7bc0ea-kube-api-access-fdpkt\") pod \"authorino-557f876f69-8zvx2\" (UID: \"194cb818-d0b5-4c4d-b0de-293e7d7bc0ea\") " pod="kuadrant-system/authorino-557f876f69-8zvx2"
Sep 15 12:34:55.784631 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.784598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdpkt\" (UniqueName: \"kubernetes.io/projected/194cb818-d0b5-4c4d-b0de-293e7d7bc0ea-kube-api-access-fdpkt\") pod \"authorino-557f876f69-8zvx2\" (UID: \"194cb818-d0b5-4c4d-b0de-293e7d7bc0ea\") " pod="kuadrant-system/authorino-557f876f69-8zvx2"
Sep 15 12:34:55.792480 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.792454    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdpkt\" (UniqueName: \"kubernetes.io/projected/194cb818-d0b5-4c4d-b0de-293e7d7bc0ea-kube-api-access-fdpkt\") pod \"authorino-557f876f69-8zvx2\" (UID: \"194cb818-d0b5-4c4d-b0de-293e7d7bc0ea\") " pod="kuadrant-system/authorino-557f876f69-8zvx2"
Sep 15 12:34:55.856806 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.856767    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-8zvx2"]
Sep 15 12:34:55.857062 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.857044    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-8zvx2"
Sep 15 12:34:55.879115 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.879089    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-7pwdh"]
Sep 15 12:34:55.885431 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.885406    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-7pwdh"
Sep 15 12:34:55.888487 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.888455    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 12:34:55.889937 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.889913    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-7pwdh"]
Sep 15 12:34:55.911645 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.911167    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-7pwdh"]
Sep 15 12:34:55.911645 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:34:55.911439    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-kvxth tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-kvxth tls-cert]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-7pwdh" podUID="7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9"
Sep 15 12:34:55.986718 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.986635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvxth\" (UniqueName: \"kubernetes.io/projected/7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9-kube-api-access-kvxth\") pod \"authorino-6d4dfc649f-7pwdh\" (UID: \"7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9\") " pod="kuadrant-system/authorino-6d4dfc649f-7pwdh"
Sep 15 12:34:55.986718 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:55.986695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9-tls-cert\") pod \"authorino-6d4dfc649f-7pwdh\" (UID: \"7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9\") " pod="kuadrant-system/authorino-6d4dfc649f-7pwdh"
Sep 15 12:34:56.087820 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:56.087786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvxth\" (UniqueName: \"kubernetes.io/projected/7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9-kube-api-access-kvxth\") pod \"authorino-6d4dfc649f-7pwdh\" (UID: \"7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9\") " pod="kuadrant-system/authorino-6d4dfc649f-7pwdh"
Sep 15 12:34:56.087989 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:56.087863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9-tls-cert\") pod \"authorino-6d4dfc649f-7pwdh\" (UID: \"7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9\") " pod="kuadrant-system/authorino-6d4dfc649f-7pwdh"
Sep 15 12:34:56.090224 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:56.090199    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9-tls-cert\") pod \"authorino-6d4dfc649f-7pwdh\" (UID: \"7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9\") " pod="kuadrant-system/authorino-6d4dfc649f-7pwdh"
Sep 15 12:34:56.096505 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:56.096481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvxth\" (UniqueName: \"kubernetes.io/projected/7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9-kube-api-access-kvxth\") pod \"authorino-6d4dfc649f-7pwdh\" (UID: \"7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9\") " pod="kuadrant-system/authorino-6d4dfc649f-7pwdh"
Sep 15 12:34:56.131126 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:56.131100    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-8zvx2"]
Sep 15 12:34:56.133153 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:34:56.133116    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod194cb818_d0b5_4c4d_b0de_293e7d7bc0ea.slice/crio-56e038aaadab58fbaf7d2ea6b4a589c0c0b07f95dca8579a194ea5a374bf760d WatchSource:0}: Error finding container 56e038aaadab58fbaf7d2ea6b4a589c0c0b07f95dca8579a194ea5a374bf760d: Status 404 returned error can't find the container with id 56e038aaadab58fbaf7d2ea6b4a589c0c0b07f95dca8579a194ea5a374bf760d
Sep 15 12:34:56.589044 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:56.589016    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-7pwdh"
Sep 15 12:34:56.589470 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:56.589018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-8zvx2" event={"ID":"194cb818-d0b5-4c4d-b0de-293e7d7bc0ea","Type":"ContainerStarted","Data":"56e038aaadab58fbaf7d2ea6b4a589c0c0b07f95dca8579a194ea5a374bf760d"}
Sep 15 12:34:56.595967 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:56.595949    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-7pwdh"
Sep 15 12:34:56.692688 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:56.692658    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvxth\" (UniqueName: \"kubernetes.io/projected/7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9-kube-api-access-kvxth\") pod \"7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9\" (UID: \"7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9\") "
Sep 15 12:34:56.692849 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:56.692721    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9-tls-cert\") pod \"7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9\" (UID: \"7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9\") "
Sep 15 12:34:56.694808 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:56.694780    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9" (UID: "7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:34:56.694945 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:56.694840    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9-kube-api-access-kvxth" (OuterVolumeSpecName: "kube-api-access-kvxth") pod "7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9" (UID: "7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9"). InnerVolumeSpecName "kube-api-access-kvxth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:34:56.793627 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:56.793602    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9-tls-cert\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:34:56.793627 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:56.793626    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvxth\" (UniqueName: \"kubernetes.io/projected/7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9-kube-api-access-kvxth\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:34:57.595499 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:57.595404    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-8zvx2" event={"ID":"194cb818-d0b5-4c4d-b0de-293e7d7bc0ea","Type":"ContainerStarted","Data":"4e1c45be025d1ce08e3379b70131801d251723f76e469e022bd0cc8dcc9c42e7"}
Sep 15 12:34:57.595499 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:57.595432    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-8zvx2" podUID="194cb818-d0b5-4c4d-b0de-293e7d7bc0ea" containerName="authorino" containerID="cri-o://4e1c45be025d1ce08e3379b70131801d251723f76e469e022bd0cc8dcc9c42e7" gracePeriod=30
Sep 15 12:34:57.595499 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:57.595466    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-7pwdh"
Sep 15 12:34:57.609301 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:57.609262    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-8zvx2" podStartSLOduration=1.974829374 podStartE2EDuration="2.609247503s" totalImagesPullingTime="634.418129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:34:57.60850438 +0000 UTC m=+568.993215265" watchObservedRunningTime="2026-09-15 12:34:57.609247503 +0000 UTC m=+568.993958387"
Sep 15 12:34:57.630819 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:57.630794    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-7pwdh"]
Sep 15 12:34:57.638779 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:57.638758    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-7pwdh"]
Sep 15 12:34:57.670367 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:57.670336    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:34:57.843697 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:57.843675    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-8zvx2"
Sep 15 12:34:57.901758 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:57.901730    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdpkt\" (UniqueName: \"kubernetes.io/projected/194cb818-d0b5-4c4d-b0de-293e7d7bc0ea-kube-api-access-fdpkt\") pod \"194cb818-d0b5-4c4d-b0de-293e7d7bc0ea\" (UID: \"194cb818-d0b5-4c4d-b0de-293e7d7bc0ea\") "
Sep 15 12:34:57.903722 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:57.903694    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194cb818-d0b5-4c4d-b0de-293e7d7bc0ea-kube-api-access-fdpkt" (OuterVolumeSpecName: "kube-api-access-fdpkt") pod "194cb818-d0b5-4c4d-b0de-293e7d7bc0ea" (UID: "194cb818-d0b5-4c4d-b0de-293e7d7bc0ea"). InnerVolumeSpecName "kube-api-access-fdpkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:34:58.002815 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:58.002787    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdpkt\" (UniqueName: \"kubernetes.io/projected/194cb818-d0b5-4c4d-b0de-293e7d7bc0ea-kube-api-access-fdpkt\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:34:58.601873 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:58.601811    2579 generic.go:356] "Generic (PLEG): container finished" podID="194cb818-d0b5-4c4d-b0de-293e7d7bc0ea" containerID="4e1c45be025d1ce08e3379b70131801d251723f76e469e022bd0cc8dcc9c42e7" exitCode=0
Sep 15 12:34:58.602287 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:58.601880    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-8zvx2"
Sep 15 12:34:58.602287 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:58.601878    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-8zvx2" event={"ID":"194cb818-d0b5-4c4d-b0de-293e7d7bc0ea","Type":"ContainerDied","Data":"4e1c45be025d1ce08e3379b70131801d251723f76e469e022bd0cc8dcc9c42e7"}
Sep 15 12:34:58.602287 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:58.601959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-8zvx2" event={"ID":"194cb818-d0b5-4c4d-b0de-293e7d7bc0ea","Type":"ContainerDied","Data":"56e038aaadab58fbaf7d2ea6b4a589c0c0b07f95dca8579a194ea5a374bf760d"}
Sep 15 12:34:58.602287 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:58.601985    2579 scope.go:117] "RemoveContainer" containerID="4e1c45be025d1ce08e3379b70131801d251723f76e469e022bd0cc8dcc9c42e7"
Sep 15 12:34:58.611619 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:58.611603    2579 scope.go:117] "RemoveContainer" containerID="4e1c45be025d1ce08e3379b70131801d251723f76e469e022bd0cc8dcc9c42e7"
Sep 15 12:34:58.611908 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:34:58.611888    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1c45be025d1ce08e3379b70131801d251723f76e469e022bd0cc8dcc9c42e7\": container with ID starting with 4e1c45be025d1ce08e3379b70131801d251723f76e469e022bd0cc8dcc9c42e7 not found: ID does not exist" containerID="4e1c45be025d1ce08e3379b70131801d251723f76e469e022bd0cc8dcc9c42e7"
Sep 15 12:34:58.611976 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:58.611917    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1c45be025d1ce08e3379b70131801d251723f76e469e022bd0cc8dcc9c42e7"} err="failed to get container status \"4e1c45be025d1ce08e3379b70131801d251723f76e469e022bd0cc8dcc9c42e7\": rpc error: code = NotFound desc = could not find container \"4e1c45be025d1ce08e3379b70131801d251723f76e469e022bd0cc8dcc9c42e7\": container with ID starting with 4e1c45be025d1ce08e3379b70131801d251723f76e469e022bd0cc8dcc9c42e7 not found: ID does not exist"
Sep 15 12:34:58.623656 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:58.623635    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-8zvx2"]
Sep 15 12:34:58.625978 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:58.625960    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-8zvx2"]
Sep 15 12:34:59.108995 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:59.108967    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="194cb818-d0b5-4c4d-b0de-293e7d7bc0ea" path="/var/lib/kubelet/pods/194cb818-d0b5-4c4d-b0de-293e7d7bc0ea/volumes"
Sep 15 12:34:59.109370 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:34:59.109354    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9" path="/var/lib/kubelet/pods/7e0a5e0f-a2ac-498d-ae79-2f3afb41d8f9/volumes"
Sep 15 12:35:01.993210 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:01.993173    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:03.572440 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:03.572402    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:06.083942 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:06.083905    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:06.182124 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:06.182088    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:29.082130 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:29.082101    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:35:29.091527 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:29.091502    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:35:37.626509 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:37.626456    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:37.716199 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:37.716164    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:39.227324 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:39.227265    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:39.325125 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:39.325079    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:39.574285 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:39.574141    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:39.705045 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:39.705009    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:39.803643 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:39.803603    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:39.866621 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:39.866593    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:35:51.432165 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:51.432129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:35:51.539299 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:35:51.539228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:02.454441 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.454406    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"]
Sep 15 12:36:02.454807 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.454741    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="194cb818-d0b5-4c4d-b0de-293e7d7bc0ea" containerName="authorino"
Sep 15 12:36:02.454807 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.454756    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="194cb818-d0b5-4c4d-b0de-293e7d7bc0ea" containerName="authorino"
Sep 15 12:36:02.454895 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.454811    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="194cb818-d0b5-4c4d-b0de-293e7d7bc0ea" containerName="authorino"
Sep 15 12:36:02.457140 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.457124    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.460777 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.460753    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 12:36:02.460921 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.460780    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2598m\""
Sep 15 12:36:02.460921 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.460800    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 15 12:36:02.460921 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.460891    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 12:36:02.467959 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.467939    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"]
Sep 15 12:36:02.493715 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.493683    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:02.532938 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.532908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b90415b-85bf-4314-96be-eae0f3af4e03-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.533079 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.532948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b90415b-85bf-4314-96be-eae0f3af4e03-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.533079 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.533032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b90415b-85bf-4314-96be-eae0f3af4e03-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.533161 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.533091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4l6s\" (UniqueName: \"kubernetes.io/projected/9b90415b-85bf-4314-96be-eae0f3af4e03-kube-api-access-r4l6s\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.533161 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.533123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b90415b-85bf-4314-96be-eae0f3af4e03-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.533230 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.533173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b90415b-85bf-4314-96be-eae0f3af4e03-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.634286 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.634254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b90415b-85bf-4314-96be-eae0f3af4e03-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.634421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.634294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b90415b-85bf-4314-96be-eae0f3af4e03-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.634421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.634333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b90415b-85bf-4314-96be-eae0f3af4e03-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.634421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.634359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4l6s\" (UniqueName: \"kubernetes.io/projected/9b90415b-85bf-4314-96be-eae0f3af4e03-kube-api-access-r4l6s\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.634421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.634379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b90415b-85bf-4314-96be-eae0f3af4e03-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.634421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.634401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b90415b-85bf-4314-96be-eae0f3af4e03-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.634766 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.634740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b90415b-85bf-4314-96be-eae0f3af4e03-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.634766 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.634759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b90415b-85bf-4314-96be-eae0f3af4e03-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.634906 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.634871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b90415b-85bf-4314-96be-eae0f3af4e03-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.636539 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.636514    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b90415b-85bf-4314-96be-eae0f3af4e03-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.636775 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.636760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b90415b-85bf-4314-96be-eae0f3af4e03-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.641635 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.641619    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4l6s\" (UniqueName: \"kubernetes.io/projected/9b90415b-85bf-4314-96be-eae0f3af4e03-kube-api-access-r4l6s\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg\" (UID: \"9b90415b-85bf-4314-96be-eae0f3af4e03\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.769344 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.769263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:02.963939 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:02.963912    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:36:03.040496 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:03.040467    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"]
Sep 15 12:36:03.042740 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:36:03.042714    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b90415b_85bf_4314_96be_eae0f3af4e03.slice/crio-f19b5e6e1b2d0386d046b8efc0073e9e87e06ef2a89429995d8f88aa1ad39145 WatchSource:0}: Error finding container f19b5e6e1b2d0386d046b8efc0073e9e87e06ef2a89429995d8f88aa1ad39145: Status 404 returned error can't find the container with id f19b5e6e1b2d0386d046b8efc0073e9e87e06ef2a89429995d8f88aa1ad39145
Sep 15 12:36:03.044079 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:03.044062    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:36:03.919054 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:03.919011    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg" event={"ID":"9b90415b-85bf-4314-96be-eae0f3af4e03","Type":"ContainerStarted","Data":"f19b5e6e1b2d0386d046b8efc0073e9e87e06ef2a89429995d8f88aa1ad39145"}
Sep 15 12:36:04.924936 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:04.924899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg" event={"ID":"9b90415b-85bf-4314-96be-eae0f3af4e03","Type":"ContainerStarted","Data":"26f2cec22b42c1ff8d3e67c49ee9d7fd11907f397692ed2d98b3937702a1a78b"}
Sep 15 12:36:04.946198 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:04.944257    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg" podStartSLOduration=1.941100791 podStartE2EDuration="2.944234432s" totalImagesPullingTime="1.003133641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:36:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:36:04.941563926 +0000 UTC m=+636.326274812" watchObservedRunningTime="2026-09-15 12:36:04.944234432 +0000 UTC m=+636.328945319"
Sep 15 12:36:05.627889 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:05.627845    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:06.252386 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:06.252348    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:36:12.769573 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:12.769533    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:12.770093 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:12.769817    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:12.781935 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:12.781908    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:12.975971 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:12.975939    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg"
Sep 15 12:36:15.188154 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:15.188129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:16.048051 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:16.048010    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:36:20.614884 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:20.614852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:21.752310 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:21.752279    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:36:25.547364 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.547330    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"]
Sep 15 12:36:25.550670 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.550654    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.553369 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.553348    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 15 12:36:25.560614 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.560593    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"]
Sep 15 12:36:25.575240 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.575205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:25.631043 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.631011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d744t\" (UniqueName: \"kubernetes.io/projected/69ff1e20-76e5-4178-b1e1-dd9858f21baf-kube-api-access-d744t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.631179 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.631056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69ff1e20-76e5-4178-b1e1-dd9858f21baf-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.631179 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.631126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69ff1e20-76e5-4178-b1e1-dd9858f21baf-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.631179 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.631147    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69ff1e20-76e5-4178-b1e1-dd9858f21baf-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.631279 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.631196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69ff1e20-76e5-4178-b1e1-dd9858f21baf-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.631279 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.631213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69ff1e20-76e5-4178-b1e1-dd9858f21baf-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.732400 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.732360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d744t\" (UniqueName: \"kubernetes.io/projected/69ff1e20-76e5-4178-b1e1-dd9858f21baf-kube-api-access-d744t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.732594 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.732421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69ff1e20-76e5-4178-b1e1-dd9858f21baf-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.732594 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.732460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69ff1e20-76e5-4178-b1e1-dd9858f21baf-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.732594 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.732487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69ff1e20-76e5-4178-b1e1-dd9858f21baf-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.732594 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.732529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69ff1e20-76e5-4178-b1e1-dd9858f21baf-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.732869 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.732660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69ff1e20-76e5-4178-b1e1-dd9858f21baf-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.732869 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.732808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69ff1e20-76e5-4178-b1e1-dd9858f21baf-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.732966 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.732910    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69ff1e20-76e5-4178-b1e1-dd9858f21baf-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.732966 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.732940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69ff1e20-76e5-4178-b1e1-dd9858f21baf-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.734709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.734687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69ff1e20-76e5-4178-b1e1-dd9858f21baf-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.735054 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.735036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69ff1e20-76e5-4178-b1e1-dd9858f21baf-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.740977 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.740949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d744t\" (UniqueName: \"kubernetes.io/projected/69ff1e20-76e5-4178-b1e1-dd9858f21baf-kube-api-access-d744t\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v\" (UID: \"69ff1e20-76e5-4178-b1e1-dd9858f21baf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:25.861535 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:25.861501    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:26.133553 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:26.133523    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"]
Sep 15 12:36:26.135117 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:36:26.135090    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ff1e20_76e5_4178_b1e1_dd9858f21baf.slice/crio-c8acd79a870a27d4914182c441643f59d1320720ca2825b2a4b4a7c076e6f8f1 WatchSource:0}: Error finding container c8acd79a870a27d4914182c441643f59d1320720ca2825b2a4b4a7c076e6f8f1: Status 404 returned error can't find the container with id c8acd79a870a27d4914182c441643f59d1320720ca2825b2a4b4a7c076e6f8f1
Sep 15 12:36:27.043373 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:27.043332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v" event={"ID":"69ff1e20-76e5-4178-b1e1-dd9858f21baf","Type":"ContainerStarted","Data":"98bfe8fa66e6724fcdff4170153ad14df37e2c910d796db3f251f48f01ccc68c"}
Sep 15 12:36:27.043752 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:27.043386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v" event={"ID":"69ff1e20-76e5-4178-b1e1-dd9858f21baf","Type":"ContainerStarted","Data":"c8acd79a870a27d4914182c441643f59d1320720ca2825b2a4b4a7c076e6f8f1"}
Sep 15 12:36:27.060823 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:27.060771    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v" podStartSLOduration=1.8228702220000002 podStartE2EDuration="2.060754021s" totalImagesPullingTime="237.883799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:36:27.059820982 +0000 UTC m=+658.444531867" watchObservedRunningTime="2026-09-15 12:36:27.060754021 +0000 UTC m=+658.445464905"
Sep 15 12:36:27.655340 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:27.655305    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:36:35.861923 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:35.861883    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:35.862365 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:35.861948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:35.874523 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:35.874492    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:36.097754 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:36.097722    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v"
Sep 15 12:36:37.760034 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:37.759995    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:39.653044 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:39.653012    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:36:46.887193 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:46.887147    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:51.056200 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.056165    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"]
Sep 15 12:36:51.061579 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.061557    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.063982 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.063961    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 15 12:36:51.068895 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.068872    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"]
Sep 15 12:36:51.081300 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.081264    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:36:51.133741 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.133714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9cb419d-9724-42b0-ac60-e05b675ce950-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.133908 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.133755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9cb419d-9724-42b0-ac60-e05b675ce950-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.133908 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.133871    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9cb419d-9724-42b0-ac60-e05b675ce950-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.134028 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.133906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9cb419d-9724-42b0-ac60-e05b675ce950-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.134028 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.133941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvrh\" (UniqueName: \"kubernetes.io/projected/e9cb419d-9724-42b0-ac60-e05b675ce950-kube-api-access-8gvrh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.134028 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.133983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9cb419d-9724-42b0-ac60-e05b675ce950-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.234730 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.234696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9cb419d-9724-42b0-ac60-e05b675ce950-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.234873 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.234747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9cb419d-9724-42b0-ac60-e05b675ce950-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.234873 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.234781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9cb419d-9724-42b0-ac60-e05b675ce950-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.234873 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.234799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9cb419d-9724-42b0-ac60-e05b675ce950-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.234873 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.234819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvrh\" (UniqueName: \"kubernetes.io/projected/e9cb419d-9724-42b0-ac60-e05b675ce950-kube-api-access-8gvrh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.235060 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.234875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9cb419d-9724-42b0-ac60-e05b675ce950-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.235255 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.235229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9cb419d-9724-42b0-ac60-e05b675ce950-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.235371 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.235286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9cb419d-9724-42b0-ac60-e05b675ce950-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.235371 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.235313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9cb419d-9724-42b0-ac60-e05b675ce950-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.237039 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.237016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9cb419d-9724-42b0-ac60-e05b675ce950-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.237336 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.237319    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9cb419d-9724-42b0-ac60-e05b675ce950-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.242372 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.242349    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvrh\" (UniqueName: \"kubernetes.io/projected/e9cb419d-9724-42b0-ac60-e05b675ce950-kube-api-access-8gvrh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc\" (UID: \"e9cb419d-9724-42b0-ac60-e05b675ce950\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.371636 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.371607    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:36:51.641457 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:51.641428    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"]
Sep 15 12:36:51.643302 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:36:51.643273    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9cb419d_9724_42b0_ac60_e05b675ce950.slice/crio-f4d45f15598a0d06e3b5f067d3ea6dacf04d7ddb6b533b531b58688969b3ef6c WatchSource:0}: Error finding container f4d45f15598a0d06e3b5f067d3ea6dacf04d7ddb6b533b531b58688969b3ef6c: Status 404 returned error can't find the container with id f4d45f15598a0d06e3b5f067d3ea6dacf04d7ddb6b533b531b58688969b3ef6c
Sep 15 12:36:52.170289 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:52.170204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc" event={"ID":"e9cb419d-9724-42b0-ac60-e05b675ce950","Type":"ContainerStarted","Data":"718384d4ffcac679c6a1ce2d19aa5bb6d331f97241c82be0093a192bff7a1e65"}
Sep 15 12:36:52.170289 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:52.170243    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc" event={"ID":"e9cb419d-9724-42b0-ac60-e05b675ce950","Type":"ContainerStarted","Data":"f4d45f15598a0d06e3b5f067d3ea6dacf04d7ddb6b533b531b58688969b3ef6c"}
Sep 15 12:36:52.188664 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:52.188618    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc" podStartSLOduration=0.951572201 podStartE2EDuration="1.188604749s" totalImagesPullingTime="237.032548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:36:52.186480196 +0000 UTC m=+683.571191079" watchObservedRunningTime="2026-09-15 12:36:52.188604749 +0000 UTC m=+683.573315632"
Sep 15 12:36:53.256227 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:36:53.256197    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:37:01.372528 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:37:01.372480    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:37:01.372947 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:37:01.372543    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:37:01.384975 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:37:01.384947    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:37:02.227323 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:37:02.227295    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc"
Sep 15 12:38:01.047893 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:01.047803    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66cfddbbb4-hnrps"]
Sep 15 12:38:01.051036 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:01.051021    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66cfddbbb4-hnrps"
Sep 15 12:38:01.054511 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:01.054490    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 12:38:01.054511 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:01.054491    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jpchf\""
Sep 15 12:38:01.059320 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:01.059293    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66cfddbbb4-hnrps"]
Sep 15 12:38:01.094957 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:01.094920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqfrc\" (UniqueName: \"kubernetes.io/projected/c09f66d1-d1b7-40f3-bde9-3d226721da78-kube-api-access-zqfrc\") pod \"authorino-66cfddbbb4-hnrps\" (UID: \"c09f66d1-d1b7-40f3-bde9-3d226721da78\") " pod="kuadrant-system/authorino-66cfddbbb4-hnrps"
Sep 15 12:38:01.095114 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:01.095002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c09f66d1-d1b7-40f3-bde9-3d226721da78-tls-cert\") pod \"authorino-66cfddbbb4-hnrps\" (UID: \"c09f66d1-d1b7-40f3-bde9-3d226721da78\") " pod="kuadrant-system/authorino-66cfddbbb4-hnrps"
Sep 15 12:38:01.195492 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:01.195456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqfrc\" (UniqueName: \"kubernetes.io/projected/c09f66d1-d1b7-40f3-bde9-3d226721da78-kube-api-access-zqfrc\") pod \"authorino-66cfddbbb4-hnrps\" (UID: \"c09f66d1-d1b7-40f3-bde9-3d226721da78\") " pod="kuadrant-system/authorino-66cfddbbb4-hnrps"
Sep 15 12:38:01.195667 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:01.195507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c09f66d1-d1b7-40f3-bde9-3d226721da78-tls-cert\") pod \"authorino-66cfddbbb4-hnrps\" (UID: \"c09f66d1-d1b7-40f3-bde9-3d226721da78\") " pod="kuadrant-system/authorino-66cfddbbb4-hnrps"
Sep 15 12:38:01.197970 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:01.197947    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c09f66d1-d1b7-40f3-bde9-3d226721da78-tls-cert\") pod \"authorino-66cfddbbb4-hnrps\" (UID: \"c09f66d1-d1b7-40f3-bde9-3d226721da78\") " pod="kuadrant-system/authorino-66cfddbbb4-hnrps"
Sep 15 12:38:01.204114 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:01.204089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqfrc\" (UniqueName: \"kubernetes.io/projected/c09f66d1-d1b7-40f3-bde9-3d226721da78-kube-api-access-zqfrc\") pod \"authorino-66cfddbbb4-hnrps\" (UID: \"c09f66d1-d1b7-40f3-bde9-3d226721da78\") " pod="kuadrant-system/authorino-66cfddbbb4-hnrps"
Sep 15 12:38:01.361051 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:01.361023    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66cfddbbb4-hnrps"
Sep 15 12:38:01.629695 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:01.629668    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66cfddbbb4-hnrps"]
Sep 15 12:38:01.631676 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:38:01.631644    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc09f66d1_d1b7_40f3_bde9_3d226721da78.slice/crio-07f994a8ce52defdd4c6870db0a77bd56ac8382a25bc6b3f135174dd8fa3b848 WatchSource:0}: Error finding container 07f994a8ce52defdd4c6870db0a77bd56ac8382a25bc6b3f135174dd8fa3b848: Status 404 returned error can't find the container with id 07f994a8ce52defdd4c6870db0a77bd56ac8382a25bc6b3f135174dd8fa3b848
Sep 15 12:38:02.519188 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:02.519152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66cfddbbb4-hnrps" event={"ID":"c09f66d1-d1b7-40f3-bde9-3d226721da78","Type":"ContainerStarted","Data":"07f994a8ce52defdd4c6870db0a77bd56ac8382a25bc6b3f135174dd8fa3b848"}
Sep 15 12:38:03.526900 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:03.526855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66cfddbbb4-hnrps" event={"ID":"c09f66d1-d1b7-40f3-bde9-3d226721da78","Type":"ContainerStarted","Data":"44bef47a4efeba72d1b200919ddd96d50befabf8c6222c4c7fa011a3924bd14a"}
Sep 15 12:38:03.548446 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:03.548397    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-66cfddbbb4-hnrps" podStartSLOduration=1.307811243 podStartE2EDuration="2.548381447s" totalImagesPullingTime="1.240570204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:38:03.54677173 +0000 UTC m=+754.931482613" watchObservedRunningTime="2026-09-15 12:38:03.548381447 +0000 UTC m=+754.933092331"
Sep 15 12:38:04.106196 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:04.106138    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:04.202135 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:04.202081    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:05.202383 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:05.202341    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:05.302503 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:05.302466    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:15.998262 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:15.998220    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:16.049798 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:16.049763    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:24.640978 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:24.640938    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:24.740918 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:24.740882    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:24.841478 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:24.841439    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:24.980735 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:24.980656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:25.070594 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:25.070558    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:26.871160 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:26.871125    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7719-c86cce"]
Sep 15 12:38:26.874339 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:26.874320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-c86cce"
Sep 15 12:38:26.877131 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:26.877107    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dhqjr\""
Sep 15 12:38:26.879085 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:26.879058    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-c86cce"]
Sep 15 12:38:26.913807 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:26.913781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqq9x\" (UniqueName: \"kubernetes.io/projected/7b446fbe-d4b8-47cb-95d4-b4103eefd25b-kube-api-access-gqq9x\") pod \"test-curl-7719-c86cce\" (UID: \"7b446fbe-d4b8-47cb-95d4-b4103eefd25b\") " pod="opendatahub/test-curl-7719-c86cce"
Sep 15 12:38:27.014357 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:27.014323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqq9x\" (UniqueName: \"kubernetes.io/projected/7b446fbe-d4b8-47cb-95d4-b4103eefd25b-kube-api-access-gqq9x\") pod \"test-curl-7719-c86cce\" (UID: \"7b446fbe-d4b8-47cb-95d4-b4103eefd25b\") " pod="opendatahub/test-curl-7719-c86cce"
Sep 15 12:38:27.022606 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:27.022580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqq9x\" (UniqueName: \"kubernetes.io/projected/7b446fbe-d4b8-47cb-95d4-b4103eefd25b-kube-api-access-gqq9x\") pod \"test-curl-7719-c86cce\" (UID: \"7b446fbe-d4b8-47cb-95d4-b4103eefd25b\") " pod="opendatahub/test-curl-7719-c86cce"
Sep 15 12:38:27.184965 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:27.184887    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-c86cce"
Sep 15 12:38:27.296694 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:27.296664    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:27.397844 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:27.397804    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:27.457879 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:27.457857    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-c86cce"]
Sep 15 12:38:27.460072 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:38:27.460042    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b446fbe_d4b8_47cb_95d4_b4103eefd25b.slice/crio-2ca89b2f577973df892ff0a91889e075e0dc1dfd160f9b8477504e0a733abff2 WatchSource:0}: Error finding container 2ca89b2f577973df892ff0a91889e075e0dc1dfd160f9b8477504e0a733abff2: Status 404 returned error can't find the container with id 2ca89b2f577973df892ff0a91889e075e0dc1dfd160f9b8477504e0a733abff2
Sep 15 12:38:27.644082 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:27.644042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-c86cce" event={"ID":"7b446fbe-d4b8-47cb-95d4-b4103eefd25b","Type":"ContainerStarted","Data":"2ca89b2f577973df892ff0a91889e075e0dc1dfd160f9b8477504e0a733abff2"}
Sep 15 12:38:27.977331 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:27.977263    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:28.074499 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:28.074453    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:28.217984 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:28.217947    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:28.334423 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:28.334325    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:28.474522 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:28.474475    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:28.762347 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:28.762309    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:38:29.398267 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:29.398228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:29.497888 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:29.497858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:29.598535 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:29.598505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:29.657633 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:29.657552    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-c86cce" event={"ID":"7b446fbe-d4b8-47cb-95d4-b4103eefd25b","Type":"ContainerStarted","Data":"53878104ddf066bd52a5cab24324bd4047ea33a13cba729a82b121245028c338"}
Sep 15 12:38:29.673322 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:29.673276    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7719-c86cce" podStartSLOduration=1.650524208 podStartE2EDuration="3.673259689s" totalImagesPullingTime="2.022735481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:38:29.672549601 +0000 UTC m=+781.057260485" watchObservedRunningTime="2026-09-15 12:38:29.673259689 +0000 UTC m=+781.057970572"
Sep 15 12:38:30.014075 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.013994    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:30.113049 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.113008    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:30.269298 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.269216    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7719-c86cce"]
Sep 15 12:38:30.275847 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.275800    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7719-c86cce"]
Sep 15 12:38:30.392187 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.392151    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7719-afa7ad"]
Sep 15 12:38:30.392533 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.392508    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b446fbe-d4b8-47cb-95d4-b4103eefd25b" containerName="test-curl-7719-c86cce"
Sep 15 12:38:30.392533 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.392523    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b446fbe-d4b8-47cb-95d4-b4103eefd25b" containerName="test-curl-7719-c86cce"
Sep 15 12:38:30.392733 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.392577    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b446fbe-d4b8-47cb-95d4-b4103eefd25b" containerName="test-curl-7719-c86cce"
Sep 15 12:38:30.395757 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.395727    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-afa7ad"
Sep 15 12:38:30.401504 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.401480    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-afa7ad"]
Sep 15 12:38:30.446917 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.446884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fst4\" (UniqueName: \"kubernetes.io/projected/793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6-kube-api-access-7fst4\") pod \"test-curl-7719-afa7ad\" (UID: \"793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6\") " pod="opendatahub/test-curl-7719-afa7ad"
Sep 15 12:38:30.547685 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.547587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fst4\" (UniqueName: \"kubernetes.io/projected/793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6-kube-api-access-7fst4\") pod \"test-curl-7719-afa7ad\" (UID: \"793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6\") " pod="opendatahub/test-curl-7719-afa7ad"
Sep 15 12:38:30.555451 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.555419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fst4\" (UniqueName: \"kubernetes.io/projected/793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6-kube-api-access-7fst4\") pod \"test-curl-7719-afa7ad\" (UID: \"793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6\") " pod="opendatahub/test-curl-7719-afa7ad"
Sep 15 12:38:30.662371 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.662325    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:30.662940 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:38:30.662917    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7719-c86cce" container="test-curl-7719-c86cce"
Sep 15 12:38:30.707216 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.707186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-afa7ad"
Sep 15 12:38:30.976371 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:30.976349    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-afa7ad"]
Sep 15 12:38:30.978639 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:38:30.978606    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod793f63d3_b0fa_4aaa_94e2_f956b6dbd2c6.slice/crio-e6f86f4d089d9c17f0e590e4971e4e46fdc3ecdf06eddc374f5a24ed5228b75c WatchSource:0}: Error finding container e6f86f4d089d9c17f0e590e4971e4e46fdc3ecdf06eddc374f5a24ed5228b75c: Status 404 returned error can't find the container with id e6f86f4d089d9c17f0e590e4971e4e46fdc3ecdf06eddc374f5a24ed5228b75c
Sep 15 12:38:31.381254 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:31.381208    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:31.438202 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:31.438167    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:31.538701 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:31.538671    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:31.669709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:31.669614    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-afa7ad" event={"ID":"793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6","Type":"ContainerStarted","Data":"3816f44d40c353b0ba97bd9ec5df1b7f18e1ab446ebe54988349d95b8b8da3cd"}
Sep 15 12:38:31.669709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:31.669660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-afa7ad" event={"ID":"793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6","Type":"ContainerStarted","Data":"e6f86f4d089d9c17f0e590e4971e4e46fdc3ecdf06eddc374f5a24ed5228b75c"}
Sep 15 12:38:31.669709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:31.669694    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7719-c86cce" podUID="7b446fbe-d4b8-47cb-95d4-b4103eefd25b" containerName="test-curl-7719-c86cce" containerID="cri-o://53878104ddf066bd52a5cab24324bd4047ea33a13cba729a82b121245028c338" gracePeriod=2
Sep 15 12:38:31.705856 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:31.705787    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7719-afa7ad" podStartSLOduration=1.705772726 podStartE2EDuration="1.705772726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:38:31.703811456 +0000 UTC m=+783.088522339" watchObservedRunningTime="2026-09-15 12:38:31.705772726 +0000 UTC m=+783.090483610"
Sep 15 12:38:31.981264 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:31.981172    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:32.080786 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:32.080754    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:32.333257 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:32.333173    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7719-afa7ad"]
Sep 15 12:38:32.340219 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:32.340191    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7719-afa7ad"]
Sep 15 12:38:32.394156 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:32.394124    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:32.451443 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:32.451407    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:32.676041 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:38:32.676011    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7719-afa7ad" container="test-curl-7719-afa7ad"
Sep 15 12:38:32.729618 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:32.729558    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:32.865671 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:32.865636    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:32.967280 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:32.967185    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:33.058424 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:33.058382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:33.206462 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:33.206428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:33.382877 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:33.382821    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:33.577348 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:33.577314    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:33.680528 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:33.680492    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7719-afa7ad" podUID="793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6" containerName="test-curl-7719-afa7ad" containerID="cri-o://3816f44d40c353b0ba97bd9ec5df1b7f18e1ab446ebe54988349d95b8b8da3cd" gracePeriod=2
Sep 15 12:38:33.831452 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:33.831428    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-c86cce"
Sep 15 12:38:33.979728 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:33.979643    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqq9x\" (UniqueName: \"kubernetes.io/projected/7b446fbe-d4b8-47cb-95d4-b4103eefd25b-kube-api-access-gqq9x\") pod \"7b446fbe-d4b8-47cb-95d4-b4103eefd25b\" (UID: \"7b446fbe-d4b8-47cb-95d4-b4103eefd25b\") "
Sep 15 12:38:33.981788 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:33.981765    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b446fbe-d4b8-47cb-95d4-b4103eefd25b-kube-api-access-gqq9x" (OuterVolumeSpecName: "kube-api-access-gqq9x") pod "7b446fbe-d4b8-47cb-95d4-b4103eefd25b" (UID: "7b446fbe-d4b8-47cb-95d4-b4103eefd25b"). InnerVolumeSpecName "kube-api-access-gqq9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:38:34.080493 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:34.080448    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqq9x\" (UniqueName: \"kubernetes.io/projected/7b446fbe-d4b8-47cb-95d4-b4103eefd25b-kube-api-access-gqq9x\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:38:34.266026 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:34.265939    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:34.366325 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:34.366292    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:34.466452 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:34.466419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:34.529280 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:34.529172    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:34.686484 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:34.686451    2579 generic.go:356] "Generic (PLEG): container finished" podID="7b446fbe-d4b8-47cb-95d4-b4103eefd25b" containerID="53878104ddf066bd52a5cab24324bd4047ea33a13cba729a82b121245028c338" exitCode=137
Sep 15 12:38:34.686926 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:34.686491    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-c86cce"
Sep 15 12:38:34.686926 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:34.686557    2579 scope.go:117] "RemoveContainer" containerID="53878104ddf066bd52a5cab24324bd4047ea33a13cba729a82b121245028c338"
Sep 15 12:38:34.695485 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:34.695467    2579 scope.go:117] "RemoveContainer" containerID="53878104ddf066bd52a5cab24324bd4047ea33a13cba729a82b121245028c338"
Sep 15 12:38:34.695750 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:38:34.695733    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53878104ddf066bd52a5cab24324bd4047ea33a13cba729a82b121245028c338\": container with ID starting with 53878104ddf066bd52a5cab24324bd4047ea33a13cba729a82b121245028c338 not found: ID does not exist" containerID="53878104ddf066bd52a5cab24324bd4047ea33a13cba729a82b121245028c338"
Sep 15 12:38:34.695793 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:34.695761    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53878104ddf066bd52a5cab24324bd4047ea33a13cba729a82b121245028c338"} err="failed to get container status \"53878104ddf066bd52a5cab24324bd4047ea33a13cba729a82b121245028c338\": rpc error: code = NotFound desc = could not find container \"53878104ddf066bd52a5cab24324bd4047ea33a13cba729a82b121245028c338\": container with ID starting with 53878104ddf066bd52a5cab24324bd4047ea33a13cba729a82b121245028c338 not found: ID does not exist"
Sep 15 12:38:34.697274 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:38:34.697253    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7719-c86cce\" is forbidden: User \"system:node:ip-10-0-129-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-17.ec2.internal' and this object" podUID="7b446fbe-d4b8-47cb-95d4-b4103eefd25b" pod="opendatahub/test-curl-7719-c86cce"
Sep 15 12:38:34.884400 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:34.884361    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:35.106281 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:35.106247    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b446fbe-d4b8-47cb-95d4-b4103eefd25b" path="/var/lib/kubelet/pods/7b446fbe-d4b8-47cb-95d4-b4103eefd25b/volumes"
Sep 15 12:38:35.243395 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:35.243307    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:35.343697 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:35.343661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:35.858723 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:35.858701    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-afa7ad"
Sep 15 12:38:35.997605 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:35.997514    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fst4\" (UniqueName: \"kubernetes.io/projected/793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6-kube-api-access-7fst4\") pod \"793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6\" (UID: \"793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6\") "
Sep 15 12:38:35.999576 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:35.999549    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6-kube-api-access-7fst4" (OuterVolumeSpecName: "kube-api-access-7fst4") pod "793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6" (UID: "793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6"). InnerVolumeSpecName "kube-api-access-7fst4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:38:36.092609 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:36.092573    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:36.098682 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:36.098662    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fst4\" (UniqueName: \"kubernetes.io/projected/793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6-kube-api-access-7fst4\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:38:36.284341 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:36.284260    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:36.373378 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:36.373310    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:36.698720 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:36.698687    2579 generic.go:356] "Generic (PLEG): container finished" podID="793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6" containerID="3816f44d40c353b0ba97bd9ec5df1b7f18e1ab446ebe54988349d95b8b8da3cd" exitCode=137
Sep 15 12:38:36.698977 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:36.698725    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-afa7ad"
Sep 15 12:38:36.698977 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:36.698792    2579 scope.go:117] "RemoveContainer" containerID="3816f44d40c353b0ba97bd9ec5df1b7f18e1ab446ebe54988349d95b8b8da3cd"
Sep 15 12:38:36.707904 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:36.707862    2579 scope.go:117] "RemoveContainer" containerID="3816f44d40c353b0ba97bd9ec5df1b7f18e1ab446ebe54988349d95b8b8da3cd"
Sep 15 12:38:36.708298 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:38:36.708278    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3816f44d40c353b0ba97bd9ec5df1b7f18e1ab446ebe54988349d95b8b8da3cd\": container with ID starting with 3816f44d40c353b0ba97bd9ec5df1b7f18e1ab446ebe54988349d95b8b8da3cd not found: ID does not exist" containerID="3816f44d40c353b0ba97bd9ec5df1b7f18e1ab446ebe54988349d95b8b8da3cd"
Sep 15 12:38:36.708373 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:36.708310    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3816f44d40c353b0ba97bd9ec5df1b7f18e1ab446ebe54988349d95b8b8da3cd"} err="failed to get container status \"3816f44d40c353b0ba97bd9ec5df1b7f18e1ab446ebe54988349d95b8b8da3cd\": rpc error: code = NotFound desc = could not find container \"3816f44d40c353b0ba97bd9ec5df1b7f18e1ab446ebe54988349d95b8b8da3cd\": container with ID starting with 3816f44d40c353b0ba97bd9ec5df1b7f18e1ab446ebe54988349d95b8b8da3cd not found: ID does not exist"
Sep 15 12:38:36.709389 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:38:36.709364    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7719-afa7ad\" is forbidden: User \"system:node:ip-10-0-129-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-17.ec2.internal' and this object" podUID="793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6" pod="opendatahub/test-curl-7719-afa7ad"
Sep 15 12:38:37.105735 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:37.105701    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6" path="/var/lib/kubelet/pods/793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6/volumes"
Sep 15 12:38:37.216040 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:37.215999    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:37.857964 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:37.857930    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:37.953949 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:37.953909    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:38.242178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:38.242093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:39.270461 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:39.270404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:40.392980 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.392948    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"]
Sep 15 12:38:40.393352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.393279    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6" containerName="test-curl-7719-afa7ad"
Sep 15 12:38:40.393352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.393291    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6" containerName="test-curl-7719-afa7ad"
Sep 15 12:38:40.393456 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.393376    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="793f63d3-b0fa-4aaa-94e2-f956b6dbd2c6" containerName="test-curl-7719-afa7ad"
Sep 15 12:38:40.397929 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.397906    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:40.400284 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.400255    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-bf37f4-serving-cert\""
Sep 15 12:38:40.400400 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.400292    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kqdbn\""
Sep 15 12:38:40.400400 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.400296    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 12:38:40.401334 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.401314    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:38:40.401468 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.401390    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 12:38:40.410490 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.410471    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"]
Sep 15 12:38:40.430097 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.430065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:40.529878 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.529827    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:40.537049 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.537027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40838255-e5ea-4358-8f95-64cc84217c5c-maas-api-tls\") pod \"maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:40.537164 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.537087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbkgh\" (UniqueName: \"kubernetes.io/projected/40838255-e5ea-4358-8f95-64cc84217c5c-kube-api-access-tbkgh\") pod \"maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:40.537164 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.537133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40838255-e5ea-4358-8f95-64cc84217c5c-openshift-service-ca\") pod \"maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:40.537164 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.537160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40838255-e5ea-4358-8f95-64cc84217c5c-metrics-certs\") pod \"maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:40.638535 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.638486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40838255-e5ea-4358-8f95-64cc84217c5c-openshift-service-ca\") pod \"maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:40.638729 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.638550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40838255-e5ea-4358-8f95-64cc84217c5c-metrics-certs\") pod \"maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:40.638729 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.638599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40838255-e5ea-4358-8f95-64cc84217c5c-maas-api-tls\") pod \"maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:40.638729 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.638662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbkgh\" (UniqueName: \"kubernetes.io/projected/40838255-e5ea-4358-8f95-64cc84217c5c-kube-api-access-tbkgh\") pod \"maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:40.638941 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:38:40.638729    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-serving-cert: secret "maas-api-e2e-autores-bf37f4-serving-cert" not found
Sep 15 12:38:40.638941 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:38:40.638799    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40838255-e5ea-4358-8f95-64cc84217c5c-maas-api-tls podName:40838255-e5ea-4358-8f95-64cc84217c5c nodeName:}" failed. No retries permitted until 2026-09-15 12:38:41.138782405 +0000 UTC m=+792.523493267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40838255-e5ea-4358-8f95-64cc84217c5c-maas-api-tls") pod "maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c" (UID: "40838255-e5ea-4358-8f95-64cc84217c5c") : secret "maas-api-e2e-autores-bf37f4-serving-cert" not found
Sep 15 12:38:40.639344 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.639323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40838255-e5ea-4358-8f95-64cc84217c5c-openshift-service-ca\") pod \"maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:40.641003 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.640984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40838255-e5ea-4358-8f95-64cc84217c5c-metrics-certs\") pod \"maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:40.646630 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.646584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbkgh\" (UniqueName: \"kubernetes.io/projected/40838255-e5ea-4358-8f95-64cc84217c5c-kube-api-access-tbkgh\") pod \"maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:40.777896 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:40.777859    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:41.072987 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.072914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:41.142943 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.142908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40838255-e5ea-4358-8f95-64cc84217c5c-maas-api-tls\") pod \"maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:41.145136 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.145118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40838255-e5ea-4358-8f95-64cc84217c5c-maas-api-tls\") pod \"maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:41.176028 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.176000    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:41.309011 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.308977    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:41.590128 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:38:41.590078    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40838255_e5ea_4358_8f95_64cc84217c5c.slice/crio-95ffa4e72bc096d2b7eb0d4e15b003156bb3dff288b5a4c6dd816d8ceeb9755e WatchSource:0}: Error finding container 95ffa4e72bc096d2b7eb0d4e15b003156bb3dff288b5a4c6dd816d8ceeb9755e: Status 404 returned error can't find the container with id 95ffa4e72bc096d2b7eb0d4e15b003156bb3dff288b5a4c6dd816d8ceeb9755e
Sep 15 12:38:41.590791 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.590765    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"]
Sep 15 12:38:41.598428 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.598405    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"]
Sep 15 12:38:41.603168 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.603150    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:41.606742 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.606691    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1f5381de-serving-cert\""
Sep 15 12:38:41.619567 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.619544    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"]
Sep 15 12:38:41.660945 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.660914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:41.725653 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.725620    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c" event={"ID":"40838255-e5ea-4358-8f95-64cc84217c5c","Type":"ContainerStarted","Data":"95ffa4e72bc096d2b7eb0d4e15b003156bb3dff288b5a4c6dd816d8ceeb9755e"}
Sep 15 12:38:41.748002 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.747966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c282202-fe84-4605-ada0-5255fb87117f-openshift-service-ca\") pod \"maas-api-e2e-ait-1f5381de-64f599877b-7grr5\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:41.748002 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.748002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c282202-fe84-4605-ada0-5255fb87117f-metrics-certs\") pod \"maas-api-e2e-ait-1f5381de-64f599877b-7grr5\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:41.748197 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.748125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sstmz\" (UniqueName: \"kubernetes.io/projected/8c282202-fe84-4605-ada0-5255fb87117f-kube-api-access-sstmz\") pod \"maas-api-e2e-ait-1f5381de-64f599877b-7grr5\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:41.748197 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.748171    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c282202-fe84-4605-ada0-5255fb87117f-maas-api-tls\") pod \"maas-api-e2e-ait-1f5381de-64f599877b-7grr5\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:41.754099 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.754068    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:41.849312 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.849281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sstmz\" (UniqueName: \"kubernetes.io/projected/8c282202-fe84-4605-ada0-5255fb87117f-kube-api-access-sstmz\") pod \"maas-api-e2e-ait-1f5381de-64f599877b-7grr5\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:41.849471 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.849337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c282202-fe84-4605-ada0-5255fb87117f-maas-api-tls\") pod \"maas-api-e2e-ait-1f5381de-64f599877b-7grr5\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:41.849471 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.849385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c282202-fe84-4605-ada0-5255fb87117f-openshift-service-ca\") pod \"maas-api-e2e-ait-1f5381de-64f599877b-7grr5\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:41.849471 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.849408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c282202-fe84-4605-ada0-5255fb87117f-metrics-certs\") pod \"maas-api-e2e-ait-1f5381de-64f599877b-7grr5\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:41.849646 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:38:41.849526    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-serving-cert: secret "maas-api-e2e-ait-1f5381de-serving-cert" not found
Sep 15 12:38:41.849646 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:38:41.849612    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c282202-fe84-4605-ada0-5255fb87117f-maas-api-tls podName:8c282202-fe84-4605-ada0-5255fb87117f nodeName:}" failed. No retries permitted until 2026-09-15 12:38:42.349589857 +0000 UTC m=+793.734300733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c282202-fe84-4605-ada0-5255fb87117f-maas-api-tls") pod "maas-api-e2e-ait-1f5381de-64f599877b-7grr5" (UID: "8c282202-fe84-4605-ada0-5255fb87117f") : secret "maas-api-e2e-ait-1f5381de-serving-cert" not found
Sep 15 12:38:41.850098 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.850076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c282202-fe84-4605-ada0-5255fb87117f-openshift-service-ca\") pod \"maas-api-e2e-ait-1f5381de-64f599877b-7grr5\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:41.851822 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.851802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c282202-fe84-4605-ada0-5255fb87117f-metrics-certs\") pod \"maas-api-e2e-ait-1f5381de-64f599877b-7grr5\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:41.861536 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.861486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sstmz\" (UniqueName: \"kubernetes.io/projected/8c282202-fe84-4605-ada0-5255fb87117f-kube-api-access-sstmz\") pod \"maas-api-e2e-ait-1f5381de-64f599877b-7grr5\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:41.876665 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.876636    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:41.983269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:41.983231    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:42.355122 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:42.355084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c282202-fe84-4605-ada0-5255fb87117f-maas-api-tls\") pod \"maas-api-e2e-ait-1f5381de-64f599877b-7grr5\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:42.358480 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:42.358398    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c282202-fe84-4605-ada0-5255fb87117f-maas-api-tls\") pod \"maas-api-e2e-ait-1f5381de-64f599877b-7grr5\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:42.359065 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:42.359035    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:42.456381 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:42.456344    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:42.517947 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:42.517914    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:42.810445 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:42.810416    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"]
Sep 15 12:38:43.184336 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:43.184299    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:43.776635 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:38:43.776592    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c282202_fe84_4605_ada0_5255fb87117f.slice/crio-ffc5cf6b5409489d92b050a41a5d7d87d5df73f8d4852517798fab6b42d45b46 WatchSource:0}: Error finding container ffc5cf6b5409489d92b050a41a5d7d87d5df73f8d4852517798fab6b42d45b46: Status 404 returned error can't find the container with id ffc5cf6b5409489d92b050a41a5d7d87d5df73f8d4852517798fab6b42d45b46
Sep 15 12:38:44.157035 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:44.156994    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:44.752161 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:44.752126    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c" event={"ID":"40838255-e5ea-4358-8f95-64cc84217c5c","Type":"ContainerStarted","Data":"b7f4b90525e89fa184a88e59f6a41896d6246116145316c3e9979357cf4674b3"}
Sep 15 12:38:44.752372 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:44.752216    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:44.753490 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:44.753467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5" event={"ID":"8c282202-fe84-4605-ada0-5255fb87117f","Type":"ContainerStarted","Data":"ffc5cf6b5409489d92b050a41a5d7d87d5df73f8d4852517798fab6b42d45b46"}
Sep 15 12:38:44.770062 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:44.770015    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c" podStartSLOduration=2.533670459 podStartE2EDuration="4.770000643s" totalImagesPullingTime="2.236330184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:38:44.768294417 +0000 UTC m=+796.153005302" watchObservedRunningTime="2026-09-15 12:38:44.770000643 +0000 UTC m=+796.154711523"
Sep 15 12:38:45.095451 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:45.095364    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:45.879105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:45.879066    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:46.674235 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:46.674206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:46.764527 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:46.764497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5" event={"ID":"8c282202-fe84-4605-ada0-5255fb87117f","Type":"ContainerStarted","Data":"539b9ed450ee3bb4c091bcf276d4377073f3a3b142e078b906e65e8fa1708a56"}
Sep 15 12:38:46.764706 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:46.764581    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:46.782422 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:46.782372    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5" podStartSLOduration=3.3241119120000002 podStartE2EDuration="5.782356964s" totalImagesPullingTime="2.458245052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:38:46.780630165 +0000 UTC m=+798.165341046" watchObservedRunningTime="2026-09-15 12:38:46.782356964 +0000 UTC m=+798.167067848"
Sep 15 12:38:46.859696 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:46.859659    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:47.034185 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:47.034092    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:47.123699 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:47.123663    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:47.224397 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:47.224365    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:47.579712 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:47.579669    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:47.666865 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:47.666819    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:38:47.768730 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:47.768688    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:48.077713 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:48.077670    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:48.221696 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:48.221657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:48.324321 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:48.324284    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:48.467549 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:48.467512    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:48.594473 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:48.594434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:48.685701 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:48.685668    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:48.788133 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:48.788039    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:49.214159 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:49.214120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:49.320260 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:49.320224    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:49.417523 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:49.417483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:49.616662 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:49.616626    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:50.764068 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:50.764041    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:38:52.284381 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:52.284301    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:52.376736 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:52.376700    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:52.776290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:52.776258    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:38:53.089957 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:53.089872    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:53.187899 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:53.187865    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:53.481220 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:53.481186    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:53.782021 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:53.781939    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:53.882928 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:53.882899    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:54.660893 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:54.660852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:55.790929 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:55.790883    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:55.891079 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:55.891044    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:56.090424 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:56.090331    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:57.190271 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:57.190239    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:57.509843 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:57.509742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:57.604948 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:57.604909    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:58.174884 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:58.174826    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:59.156435 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:59.156406    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:59.573283 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:59.573200    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:38:59.983846 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:38:59.983797    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:00.719143 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:00.719110    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:01.213920 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.213885    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:01.594514 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.594434    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"]
Sep 15 12:39:01.597855 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.597820    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:01.601293 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.601275    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2e3fef-serving-cert\""
Sep 15 12:39:01.618382 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.618348    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"]
Sep 15 12:39:01.624715 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.624686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66a5635c-5286-47e9-8270-5e82916b4156-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:01.624854 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.624720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a5635c-5286-47e9-8270-5e82916b4156-metrics-certs\") pod \"maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:01.624924 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.624873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66a5635c-5286-47e9-8270-5e82916b4156-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:01.624924 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.624913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n7xz\" (UniqueName: \"kubernetes.io/projected/66a5635c-5286-47e9-8270-5e82916b4156-kube-api-access-6n7xz\") pod \"maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:01.630661 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.630633    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:01.687464 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.687434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:01.725753 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.725722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66a5635c-5286-47e9-8270-5e82916b4156-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:01.726122 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.725766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a5635c-5286-47e9-8270-5e82916b4156-metrics-certs\") pod \"maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:01.726122 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.725801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66a5635c-5286-47e9-8270-5e82916b4156-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:01.726122 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:39:01.725899    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-serving-cert: secret "maas-api-e2e-inf-b-2e3fef-serving-cert" not found
Sep 15 12:39:01.726122 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.725964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n7xz\" (UniqueName: \"kubernetes.io/projected/66a5635c-5286-47e9-8270-5e82916b4156-kube-api-access-6n7xz\") pod \"maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:01.726122 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:39:01.725982    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a5635c-5286-47e9-8270-5e82916b4156-maas-api-tls podName:66a5635c-5286-47e9-8270-5e82916b4156 nodeName:}" failed. No retries permitted until 2026-09-15 12:39:02.225960235 +0000 UTC m=+813.610671099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66a5635c-5286-47e9-8270-5e82916b4156-maas-api-tls") pod "maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw" (UID: "66a5635c-5286-47e9-8270-5e82916b4156") : secret "maas-api-e2e-inf-b-2e3fef-serving-cert" not found
Sep 15 12:39:01.726565 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.726546    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66a5635c-5286-47e9-8270-5e82916b4156-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:01.728213 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.728193    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a5635c-5286-47e9-8270-5e82916b4156-metrics-certs\") pod \"maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:01.737010 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:01.736987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n7xz\" (UniqueName: \"kubernetes.io/projected/66a5635c-5286-47e9-8270-5e82916b4156-kube-api-access-6n7xz\") pod \"maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:02.152614 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:02.152578    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:02.231377 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:02.231343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66a5635c-5286-47e9-8270-5e82916b4156-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:02.233775 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:02.233750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66a5635c-5286-47e9-8270-5e82916b4156-maas-api-tls\") pod \"maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:02.253059 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:02.253029    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:02.354662 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:02.354611    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:02.518538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:02.518450    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:02.789565 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:02.789543    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"]
Sep 15 12:39:02.791275 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:39:02.791249    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a5635c_5286_47e9_8270_5e82916b4156.slice/crio-66236b19ff08a9c583f35a2508e513039c448e7e3f80493c48ae95ddc50cc771 WatchSource:0}: Error finding container 66236b19ff08a9c583f35a2508e513039c448e7e3f80493c48ae95ddc50cc771: Status 404 returned error can't find the container with id 66236b19ff08a9c583f35a2508e513039c448e7e3f80493c48ae95ddc50cc771
Sep 15 12:39:02.848807 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:02.848771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw" event={"ID":"66a5635c-5286-47e9-8270-5e82916b4156","Type":"ContainerStarted","Data":"66236b19ff08a9c583f35a2508e513039c448e7e3f80493c48ae95ddc50cc771"}
Sep 15 12:39:02.929426 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:02.929390    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:03.856078 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:03.856040    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw" event={"ID":"66a5635c-5286-47e9-8270-5e82916b4156","Type":"ContainerStarted","Data":"7dacc829cd19c0218d13f7b2db564334e0ff30747aeccf2042a04aff8502eeb6"}
Sep 15 12:39:03.856488 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:03.856120    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:03.874099 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:03.874044    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw" podStartSLOduration=2.452990622 podStartE2EDuration="2.874029076s" totalImagesPullingTime="421.038454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:39:03.872187491 +0000 UTC m=+815.256898376" watchObservedRunningTime="2026-09-15 12:39:03.874029076 +0000 UTC m=+815.258739959"
Sep 15 12:39:03.911402 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:03.911369    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:04.774784 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:04.774750    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:05.609591 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:05.609557    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:06.268468 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.268432    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"]
Sep 15 12:39:06.272405 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.272389    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.275862 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.275814    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ee96ec\"/\"openshift-service-ca.crt\""
Sep 15 12:39:06.275973 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.275816    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ee96ec\"/\"simulator-w4-ee96ec-kserve-self-signed-certs\""
Sep 15 12:39:06.275973 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.275875    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ee96ec\"/\"kube-root-ca.crt\""
Sep 15 12:39:06.276115 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.276089    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ee96ec\"/\"default-dockercfg-gcf26\""
Sep 15 12:39:06.279944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.279923    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"]
Sep 15 12:39:06.364494 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.364463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-tmp-dir\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.364494 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.364499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-dshm\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.364676 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.364514    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-model-cache\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.364676 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.364539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-home\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.364676 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.364595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npktd\" (UniqueName: \"kubernetes.io/projected/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-kube-api-access-npktd\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.364676 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.364660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-tls-certs\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.395198 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.395165    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:06.465792 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.465756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-tls-certs\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.465994 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.465809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-tmp-dir\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.465994 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.465825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-dshm\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.465994 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.465865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-model-cache\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.465994 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.465885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-home\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.465994 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.465923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npktd\" (UniqueName: \"kubernetes.io/projected/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-kube-api-access-npktd\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.466374 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.466345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-home\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.466441 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.466358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-tmp-dir\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.466486 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.466449    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-model-cache\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.468132 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.468111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-dshm\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.468444 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.468423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-tls-certs\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.473747 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.473725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npktd\" (UniqueName: \"kubernetes.io/projected/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-kube-api-access-npktd\") pod \"simulator-w4-ee96ec-kserve-779b65bbc4-ssq86\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") " pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.496256 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.496228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:06.583639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.583552    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:06.853748 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.853721    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"]
Sep 15 12:39:06.855137 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:39:06.855112    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a45c3a4_8f63_4f88_ae6f_80d2ee45d492.slice/crio-be317ebc3313584514669846ee1984f449734e0c0376c29c1f2ae7fe962244a1 WatchSource:0}: Error finding container be317ebc3313584514669846ee1984f449734e0c0376c29c1f2ae7fe962244a1: Status 404 returned error can't find the container with id be317ebc3313584514669846ee1984f449734e0c0376c29c1f2ae7fe962244a1
Sep 15 12:39:06.873493 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:06.873469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86" event={"ID":"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492","Type":"ContainerStarted","Data":"be317ebc3313584514669846ee1984f449734e0c0376c29c1f2ae7fe962244a1"}
Sep 15 12:39:07.879950 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:07.879904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86" event={"ID":"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492","Type":"ContainerStarted","Data":"56395ca921ddc6b3ebf7bd1f4d38236ae9fd82a53a897ea53e9310f181037b31"}
Sep 15 12:39:07.896517 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:07.896472    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86" podStartSLOduration=1.896457523 podStartE2EDuration="1.896457523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:39:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:39:07.894989449 +0000 UTC m=+819.279700334" watchObservedRunningTime="2026-09-15 12:39:07.896457523 +0000 UTC m=+819.281168406"
Sep 15 12:39:08.578321 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.578287    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"]
Sep 15 12:39:08.578555 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.578529    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5" podUID="8c282202-fe84-4605-ada0-5255fb87117f" containerName="maas-api" containerID="cri-o://539b9ed450ee3bb4c091bcf276d4377073f3a3b142e078b906e65e8fa1708a56" gracePeriod=30
Sep 15 12:39:08.608233 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.608187    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:08.687954 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.687919    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:08.750749 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.750656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:08.820077 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.820044    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:08.822550 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.822533    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:39:08.879584 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.879547    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:08.884719 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.884698    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c282202-fe84-4605-ada0-5255fb87117f-metrics-certs\") pod \"8c282202-fe84-4605-ada0-5255fb87117f\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") "
Sep 15 12:39:08.885154 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.884763    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c282202-fe84-4605-ada0-5255fb87117f-openshift-service-ca\") pod \"8c282202-fe84-4605-ada0-5255fb87117f\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") "
Sep 15 12:39:08.885154 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.884793    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c282202-fe84-4605-ada0-5255fb87117f-maas-api-tls\") pod \"8c282202-fe84-4605-ada0-5255fb87117f\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") "
Sep 15 12:39:08.885154 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.884887    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sstmz\" (UniqueName: \"kubernetes.io/projected/8c282202-fe84-4605-ada0-5255fb87117f-kube-api-access-sstmz\") pod \"8c282202-fe84-4605-ada0-5255fb87117f\" (UID: \"8c282202-fe84-4605-ada0-5255fb87117f\") "
Sep 15 12:39:08.885323 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.885207    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c282202-fe84-4605-ada0-5255fb87117f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c282202-fe84-4605-ada0-5255fb87117f" (UID: "8c282202-fe84-4605-ada0-5255fb87117f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:39:08.886272 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.886243    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c282202-fe84-4605-ada0-5255fb87117f" containerID="539b9ed450ee3bb4c091bcf276d4377073f3a3b142e078b906e65e8fa1708a56" exitCode=0
Sep 15 12:39:08.886408 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.886292    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"
Sep 15 12:39:08.886408 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.886335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5" event={"ID":"8c282202-fe84-4605-ada0-5255fb87117f","Type":"ContainerDied","Data":"539b9ed450ee3bb4c091bcf276d4377073f3a3b142e078b906e65e8fa1708a56"}
Sep 15 12:39:08.886532 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.886404    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5" event={"ID":"8c282202-fe84-4605-ada0-5255fb87117f","Type":"ContainerDied","Data":"ffc5cf6b5409489d92b050a41a5d7d87d5df73f8d4852517798fab6b42d45b46"}
Sep 15 12:39:08.886532 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.886444    2579 scope.go:117] "RemoveContainer" containerID="539b9ed450ee3bb4c091bcf276d4377073f3a3b142e078b906e65e8fa1708a56"
Sep 15 12:39:08.887358 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.887332    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c282202-fe84-4605-ada0-5255fb87117f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c282202-fe84-4605-ada0-5255fb87117f" (UID: "8c282202-fe84-4605-ada0-5255fb87117f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:39:08.887499 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.887365    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c282202-fe84-4605-ada0-5255fb87117f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8c282202-fe84-4605-ada0-5255fb87117f" (UID: "8c282202-fe84-4605-ada0-5255fb87117f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:39:08.887499 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.887387    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c282202-fe84-4605-ada0-5255fb87117f-kube-api-access-sstmz" (OuterVolumeSpecName: "kube-api-access-sstmz") pod "8c282202-fe84-4605-ada0-5255fb87117f" (UID: "8c282202-fe84-4605-ada0-5255fb87117f"). InnerVolumeSpecName "kube-api-access-sstmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:39:08.904732 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.904712    2579 scope.go:117] "RemoveContainer" containerID="539b9ed450ee3bb4c091bcf276d4377073f3a3b142e078b906e65e8fa1708a56"
Sep 15 12:39:08.905003 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:39:08.904984    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"539b9ed450ee3bb4c091bcf276d4377073f3a3b142e078b906e65e8fa1708a56\": container with ID starting with 539b9ed450ee3bb4c091bcf276d4377073f3a3b142e078b906e65e8fa1708a56 not found: ID does not exist" containerID="539b9ed450ee3bb4c091bcf276d4377073f3a3b142e078b906e65e8fa1708a56"
Sep 15 12:39:08.905070 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.905009    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"539b9ed450ee3bb4c091bcf276d4377073f3a3b142e078b906e65e8fa1708a56"} err="failed to get container status \"539b9ed450ee3bb4c091bcf276d4377073f3a3b142e078b906e65e8fa1708a56\": rpc error: code = NotFound desc = could not find container \"539b9ed450ee3bb4c091bcf276d4377073f3a3b142e078b906e65e8fa1708a56\": container with ID starting with 539b9ed450ee3bb4c091bcf276d4377073f3a3b142e078b906e65e8fa1708a56 not found: ID does not exist"
Sep 15 12:39:08.968644 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.968606    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:08.985982 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.985953    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c282202-fe84-4605-ada0-5255fb87117f-metrics-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:39:08.985982 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.985985    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c282202-fe84-4605-ada0-5255fb87117f-openshift-service-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:39:08.986159 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.986000    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c282202-fe84-4605-ada0-5255fb87117f-maas-api-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:39:08.986159 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:08.986014    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sstmz\" (UniqueName: \"kubernetes.io/projected/8c282202-fe84-4605-ada0-5255fb87117f-kube-api-access-sstmz\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:39:09.203276 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:09.203242    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"]
Sep 15 12:39:09.209721 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:09.209686    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f5381de-64f599877b-7grr5"]
Sep 15 12:39:09.298326 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:09.298286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:09.486507 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:09.486418    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:09.643438 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:09.643401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:09.741190 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:09.741100    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:09.867358 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:09.867331    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:39:10.032135 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:10.032041    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:10.358986 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:10.358956    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:10.457001 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:10.456971    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:11.077262 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:11.077224    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:11.105143 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:11.105109    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c282202-fe84-4605-ada0-5255fb87117f" path="/var/lib/kubelet/pods/8c282202-fe84-4605-ada0-5255fb87117f/volumes"
Sep 15 12:39:11.793687 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:11.793650    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:12.303786 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:12.303730    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:12.548553 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:12.548513    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:12.690517 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:12.690474    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:13.170853 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:13.170796    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:13.271095 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:13.271062    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:13.386787 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:13.386751    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:13.664588 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:13.664552    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:39:14.279799 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:14.279765    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:15.102783 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:15.102749    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:15.895629 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:15.895592    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:16.547141 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.547109    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"]
Sep 15 12:39:16.547631 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.547616    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c282202-fe84-4605-ada0-5255fb87117f" containerName="maas-api"
Sep 15 12:39:16.547679 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.547634    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c282202-fe84-4605-ada0-5255fb87117f" containerName="maas-api"
Sep 15 12:39:16.547781 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.547770    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c282202-fe84-4605-ada0-5255fb87117f" containerName="maas-api"
Sep 15 12:39:16.558948 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.558212    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:16.561057 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.561025    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ecae662b-serving-cert\""
Sep 15 12:39:16.563716 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.563668    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"]
Sep 15 12:39:16.577712 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.577683    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:16.584451 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.584426    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:16.584555 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.584472    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:16.597424 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.597406    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:16.653775 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.653741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80cf6d87-db7d-435a-9cff-4989fdc7e48c-metrics-certs\") pod \"maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:16.653962 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.653819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80cf6d87-db7d-435a-9cff-4989fdc7e48c-openshift-service-ca\") pod \"maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:16.653962 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.653904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf6d4\" (UniqueName: \"kubernetes.io/projected/80cf6d87-db7d-435a-9cff-4989fdc7e48c-kube-api-access-bf6d4\") pod \"maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:16.653962 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.653930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80cf6d87-db7d-435a-9cff-4989fdc7e48c-maas-api-tls\") pod \"maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:16.676289 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.676255    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:16.754685 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.754657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80cf6d87-db7d-435a-9cff-4989fdc7e48c-metrics-certs\") pod \"maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:16.754855 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.754714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80cf6d87-db7d-435a-9cff-4989fdc7e48c-openshift-service-ca\") pod \"maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:16.754917 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.754898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf6d4\" (UniqueName: \"kubernetes.io/projected/80cf6d87-db7d-435a-9cff-4989fdc7e48c-kube-api-access-bf6d4\") pod \"maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:16.755009 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.754946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80cf6d87-db7d-435a-9cff-4989fdc7e48c-maas-api-tls\") pod \"maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:16.755424 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.755406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80cf6d87-db7d-435a-9cff-4989fdc7e48c-openshift-service-ca\") pod \"maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:16.757144 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.757125    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80cf6d87-db7d-435a-9cff-4989fdc7e48c-metrics-certs\") pod \"maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:16.757253 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.757176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80cf6d87-db7d-435a-9cff-4989fdc7e48c-maas-api-tls\") pod \"maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:16.762123 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.762105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf6d4\" (UniqueName: \"kubernetes.io/projected/80cf6d87-db7d-435a-9cff-4989fdc7e48c-kube-api-access-bf6d4\") pod \"maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:16.776156 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.776128    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:16.873271 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.873232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:16.947038 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:16.947002    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:39:17.075889 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:17.075858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:17.145350 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:17.145327    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"]
Sep 15 12:39:17.147548 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:39:17.147504    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cf6d87_db7d_435a_9cff_4989fdc7e48c.slice/crio-131a77d636919e2399a8b9d1be3b7fc9d2de415a8e6a86b09c852e82fac5a452 WatchSource:0}: Error finding container 131a77d636919e2399a8b9d1be3b7fc9d2de415a8e6a86b09c852e82fac5a452: Status 404 returned error can't find the container with id 131a77d636919e2399a8b9d1be3b7fc9d2de415a8e6a86b09c852e82fac5a452
Sep 15 12:39:17.147754 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:17.147728    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:17.523760 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:17.523724    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:17.937895 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:17.937860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr" event={"ID":"80cf6d87-db7d-435a-9cff-4989fdc7e48c","Type":"ContainerStarted","Data":"3c113e1dbe75ccf1dea34cdc2240933d036e8d33bc73455ede4d5e8c72055e14"}
Sep 15 12:39:17.938269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:17.937920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr" event={"ID":"80cf6d87-db7d-435a-9cff-4989fdc7e48c","Type":"ContainerStarted","Data":"131a77d636919e2399a8b9d1be3b7fc9d2de415a8e6a86b09c852e82fac5a452"}
Sep 15 12:39:17.938269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:17.938047    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:17.955709 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:17.955661    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr" podStartSLOduration=1.65424752 podStartE2EDuration="1.955646795s" totalImagesPullingTime="301.399275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:39:17.953316624 +0000 UTC m=+829.338027509" watchObservedRunningTime="2026-09-15 12:39:17.955646795 +0000 UTC m=+829.340357678"
Sep 15 12:39:18.526403 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:18.526367    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:22.766372 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:22.766335    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:23.947980 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:23.947952    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:35.157675 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:35.157641    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:39:37.519213 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:37.519182    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:38.915902 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:38.915862    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:39.682328 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:39.682286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:40.578689 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:40.578654    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:41.299662 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:41.299627    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:42.123816 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:42.123776    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:42.995094 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:42.995052    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:43.794865 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:43.794810    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:44.491589 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:44.491542    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:45.163035 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.162992    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:45.335208 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.335174    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"]
Sep 15 12:39:45.342666 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.342641    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.345721 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.345699    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-11f865\"/\"default-dockercfg-9fd5c\""
Sep 15 12:39:45.345949 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.345927    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-11f865\"/\"kube-root-ca.crt\""
Sep 15 12:39:45.346643 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.346548    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-11f865\"/\"simulator-w0-11f865-kserve-self-signed-certs\""
Sep 15 12:39:45.346826 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.346771    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-11f865\"/\"openshift-service-ca.crt\""
Sep 15 12:39:45.349192 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.348145    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"]
Sep 15 12:39:45.514331 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.514240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhhkb\" (UniqueName: \"kubernetes.io/projected/4d6d2376-d742-421c-bccf-0fcc689eb29b-kube-api-access-zhhkb\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.514331 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.514320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-model-cache\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.514569 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.514363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-dshm\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.514569 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.514386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-tmp-dir\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.514569 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.514445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-home\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.514569 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.514470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d6d2376-d742-421c-bccf-0fcc689eb29b-tls-certs\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.537726 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.537689    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:45.615222 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.615189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhhkb\" (UniqueName: \"kubernetes.io/projected/4d6d2376-d742-421c-bccf-0fcc689eb29b-kube-api-access-zhhkb\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.615429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.615253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-model-cache\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.615429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.615293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-dshm\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.615429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.615316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-tmp-dir\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.615429 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.615354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-home\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.615635 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.615486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d6d2376-d742-421c-bccf-0fcc689eb29b-tls-certs\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.615773 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.615754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-model-cache\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.615870 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.615779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-home\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.615870 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.615819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-tmp-dir\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.617499 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.617480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-dshm\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.618022 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.618001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d6d2376-d742-421c-bccf-0fcc689eb29b-tls-certs\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.622571 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.622549    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhhkb\" (UniqueName: \"kubernetes.io/projected/4d6d2376-d742-421c-bccf-0fcc689eb29b-kube-api-access-zhhkb\") pod \"simulator-w0-11f865-kserve-5b5f9c4966-2c447\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") " pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.659212 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.659184    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:45.928651 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.928623    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"]
Sep 15 12:39:45.930160 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:39:45.930126    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d6d2376_d742_421c_bccf_0fcc689eb29b.slice/crio-edaa2b387be5a5f50aeda5fca6492fe6df2ed41e4c8bfcb392749ce712e0861f WatchSource:0}: Error finding container edaa2b387be5a5f50aeda5fca6492fe6df2ed41e4c8bfcb392749ce712e0861f: Status 404 returned error can't find the container with id edaa2b387be5a5f50aeda5fca6492fe6df2ed41e4c8bfcb392749ce712e0861f
Sep 15 12:39:45.987563 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:45.987525    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:46.089491 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:46.089441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447" event={"ID":"4d6d2376-d742-421c-bccf-0fcc689eb29b","Type":"ContainerStarted","Data":"a6b84e8466c6443412f97671f1b7af3462a38f384a3b3a54b9c89e683e9e230b"}
Sep 15 12:39:46.089491 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:46.089493    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447" event={"ID":"4d6d2376-d742-421c-bccf-0fcc689eb29b","Type":"ContainerStarted","Data":"edaa2b387be5a5f50aeda5fca6492fe6df2ed41e4c8bfcb392749ce712e0861f"}
Sep 15 12:39:46.109147 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:46.109102    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447" podStartSLOduration=1.109087194 podStartE2EDuration="1.109087194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:39:46.106297654 +0000 UTC m=+857.491008538" watchObservedRunningTime="2026-09-15 12:39:46.109087194 +0000 UTC m=+857.493798076"
Sep 15 12:39:46.649080 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:46.649044    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:47.377994 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:47.377958    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:48.013157 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:48.013115    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:52.850669 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:52.850630    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:39:54.308490 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:54.308450    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:54.860473 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:54.860441    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"]
Sep 15 12:39:54.860715 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:54.860685    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr" podUID="80cf6d87-db7d-435a-9cff-4989fdc7e48c" containerName="maas-api" containerID="cri-o://3c113e1dbe75ccf1dea34cdc2240933d036e8d33bc73455ede4d5e8c72055e14" gracePeriod=30
Sep 15 12:39:54.925194 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:54.925156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:55.028526 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.028491    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:55.112916 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.112858    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:55.136876 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.136822    2579 generic.go:356] "Generic (PLEG): container finished" podID="80cf6d87-db7d-435a-9cff-4989fdc7e48c" containerID="3c113e1dbe75ccf1dea34cdc2240933d036e8d33bc73455ede4d5e8c72055e14" exitCode=0
Sep 15 12:39:55.136876 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.136871    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"
Sep 15 12:39:55.137085 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.136882    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr" event={"ID":"80cf6d87-db7d-435a-9cff-4989fdc7e48c","Type":"ContainerDied","Data":"3c113e1dbe75ccf1dea34cdc2240933d036e8d33bc73455ede4d5e8c72055e14"}
Sep 15 12:39:55.137085 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.136934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr" event={"ID":"80cf6d87-db7d-435a-9cff-4989fdc7e48c","Type":"ContainerDied","Data":"131a77d636919e2399a8b9d1be3b7fc9d2de415a8e6a86b09c852e82fac5a452"}
Sep 15 12:39:55.137085 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.136968    2579 scope.go:117] "RemoveContainer" containerID="3c113e1dbe75ccf1dea34cdc2240933d036e8d33bc73455ede4d5e8c72055e14"
Sep 15 12:39:55.147552 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.147452    2579 scope.go:117] "RemoveContainer" containerID="3c113e1dbe75ccf1dea34cdc2240933d036e8d33bc73455ede4d5e8c72055e14"
Sep 15 12:39:55.148328 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:39:55.148304    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c113e1dbe75ccf1dea34cdc2240933d036e8d33bc73455ede4d5e8c72055e14\": container with ID starting with 3c113e1dbe75ccf1dea34cdc2240933d036e8d33bc73455ede4d5e8c72055e14 not found: ID does not exist" containerID="3c113e1dbe75ccf1dea34cdc2240933d036e8d33bc73455ede4d5e8c72055e14"
Sep 15 12:39:55.148447 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.148332    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c113e1dbe75ccf1dea34cdc2240933d036e8d33bc73455ede4d5e8c72055e14"} err="failed to get container status \"3c113e1dbe75ccf1dea34cdc2240933d036e8d33bc73455ede4d5e8c72055e14\": rpc error: code = NotFound desc = could not find container \"3c113e1dbe75ccf1dea34cdc2240933d036e8d33bc73455ede4d5e8c72055e14\": container with ID starting with 3c113e1dbe75ccf1dea34cdc2240933d036e8d33bc73455ede4d5e8c72055e14 not found: ID does not exist"
Sep 15 12:39:55.177026 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.176995    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:55.200181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.200158    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80cf6d87-db7d-435a-9cff-4989fdc7e48c-maas-api-tls\") pod \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") "
Sep 15 12:39:55.200298 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.200198    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80cf6d87-db7d-435a-9cff-4989fdc7e48c-metrics-certs\") pod \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") "
Sep 15 12:39:55.200298 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.200235    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80cf6d87-db7d-435a-9cff-4989fdc7e48c-openshift-service-ca\") pod \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") "
Sep 15 12:39:55.200298 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.200262    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bf6d4\" (UniqueName: \"kubernetes.io/projected/80cf6d87-db7d-435a-9cff-4989fdc7e48c-kube-api-access-bf6d4\") pod \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\" (UID: \"80cf6d87-db7d-435a-9cff-4989fdc7e48c\") "
Sep 15 12:39:55.200620 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.200594    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80cf6d87-db7d-435a-9cff-4989fdc7e48c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "80cf6d87-db7d-435a-9cff-4989fdc7e48c" (UID: "80cf6d87-db7d-435a-9cff-4989fdc7e48c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:39:55.202320 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.202290    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80cf6d87-db7d-435a-9cff-4989fdc7e48c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "80cf6d87-db7d-435a-9cff-4989fdc7e48c" (UID: "80cf6d87-db7d-435a-9cff-4989fdc7e48c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:39:55.202761 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.202740    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80cf6d87-db7d-435a-9cff-4989fdc7e48c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "80cf6d87-db7d-435a-9cff-4989fdc7e48c" (UID: "80cf6d87-db7d-435a-9cff-4989fdc7e48c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:39:55.202901 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.202764    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80cf6d87-db7d-435a-9cff-4989fdc7e48c-kube-api-access-bf6d4" (OuterVolumeSpecName: "kube-api-access-bf6d4") pod "80cf6d87-db7d-435a-9cff-4989fdc7e48c" (UID: "80cf6d87-db7d-435a-9cff-4989fdc7e48c"). InnerVolumeSpecName "kube-api-access-bf6d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:39:55.263382 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.263320    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:55.301699 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.301658    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80cf6d87-db7d-435a-9cff-4989fdc7e48c-maas-api-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:39:55.301699 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.301691    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80cf6d87-db7d-435a-9cff-4989fdc7e48c-metrics-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:39:55.301699 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.301705    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80cf6d87-db7d-435a-9cff-4989fdc7e48c-openshift-service-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:39:55.301951 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.301718    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf6d4\" (UniqueName: \"kubernetes.io/projected/80cf6d87-db7d-435a-9cff-4989fdc7e48c-kube-api-access-bf6d4\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:39:55.360635 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.360595    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:55.458218 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.458185    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"]
Sep 15 12:39:55.464461 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.464437    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ecae662b-99cc88c48-bqqfr"]
Sep 15 12:39:55.650402 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.650357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:55.660157 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.660131    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:55.660257 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.660172    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:55.672731 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.672710    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:55.758650 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:55.758554    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:56.072777 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:56.072680    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:56.154009 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:56.153982    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:39:56.355561 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:56.355528    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:56.456102 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:56.456069    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:57.106571 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:57.106533    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80cf6d87-db7d-435a-9cff-4989fdc7e48c" path="/var/lib/kubelet/pods/80cf6d87-db7d-435a-9cff-4989fdc7e48c/volumes"
Sep 15 12:39:57.192611 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:57.192581    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:57.929301 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:57.929260    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:58.384651 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:58.384614    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:58.474371 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:58.474343    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:58.784804 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:58.784723    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:39:59.584387 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:39:59.584357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:00.249392 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.249359    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"]
Sep 15 12:40:00.249791 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.249773    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80cf6d87-db7d-435a-9cff-4989fdc7e48c" containerName="maas-api"
Sep 15 12:40:00.249904 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.249793    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="80cf6d87-db7d-435a-9cff-4989fdc7e48c" containerName="maas-api"
Sep 15 12:40:00.249904 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.249899    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="80cf6d87-db7d-435a-9cff-4989fdc7e48c" containerName="maas-api"
Sep 15 12:40:00.252781 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.252761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:00.255170 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.255148    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-81247cf5-serving-cert\""
Sep 15 12:40:00.264983 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.264959    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"]
Sep 15 12:40:00.285077 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.285044    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:00.346956 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.346928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/586eefea-3b96-4f69-ad48-6fc245743509-maas-api-tls\") pod \"maas-api-e2e-ait-81247cf5-79f5845899-wlv2v\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:00.347125 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.346971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpw7c\" (UniqueName: \"kubernetes.io/projected/586eefea-3b96-4f69-ad48-6fc245743509-kube-api-access-jpw7c\") pod \"maas-api-e2e-ait-81247cf5-79f5845899-wlv2v\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:00.347125 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.347002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/586eefea-3b96-4f69-ad48-6fc245743509-openshift-service-ca\") pod \"maas-api-e2e-ait-81247cf5-79f5845899-wlv2v\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:00.347204 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.347112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586eefea-3b96-4f69-ad48-6fc245743509-metrics-certs\") pod \"maas-api-e2e-ait-81247cf5-79f5845899-wlv2v\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:00.385762 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.385733    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:00.447934 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.447901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpw7c\" (UniqueName: \"kubernetes.io/projected/586eefea-3b96-4f69-ad48-6fc245743509-kube-api-access-jpw7c\") pod \"maas-api-e2e-ait-81247cf5-79f5845899-wlv2v\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:00.448111 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.447943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/586eefea-3b96-4f69-ad48-6fc245743509-openshift-service-ca\") pod \"maas-api-e2e-ait-81247cf5-79f5845899-wlv2v\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:00.448111 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.447976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586eefea-3b96-4f69-ad48-6fc245743509-metrics-certs\") pod \"maas-api-e2e-ait-81247cf5-79f5845899-wlv2v\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:00.448111 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.448054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/586eefea-3b96-4f69-ad48-6fc245743509-maas-api-tls\") pod \"maas-api-e2e-ait-81247cf5-79f5845899-wlv2v\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:00.448287 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:40:00.448146    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-serving-cert: secret "maas-api-e2e-ait-81247cf5-serving-cert" not found
Sep 15 12:40:00.448287 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:40:00.448212    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/586eefea-3b96-4f69-ad48-6fc245743509-maas-api-tls podName:586eefea-3b96-4f69-ad48-6fc245743509 nodeName:}" failed. No retries permitted until 2026-09-15 12:40:00.948193941 +0000 UTC m=+872.332904802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/586eefea-3b96-4f69-ad48-6fc245743509-maas-api-tls") pod "maas-api-e2e-ait-81247cf5-79f5845899-wlv2v" (UID: "586eefea-3b96-4f69-ad48-6fc245743509") : secret "maas-api-e2e-ait-81247cf5-serving-cert" not found
Sep 15 12:40:00.448672 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.448650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/586eefea-3b96-4f69-ad48-6fc245743509-openshift-service-ca\") pod \"maas-api-e2e-ait-81247cf5-79f5845899-wlv2v\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:00.450558 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.450523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586eefea-3b96-4f69-ad48-6fc245743509-metrics-certs\") pod \"maas-api-e2e-ait-81247cf5-79f5845899-wlv2v\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:00.458375 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.458351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpw7c\" (UniqueName: \"kubernetes.io/projected/586eefea-3b96-4f69-ad48-6fc245743509-kube-api-access-jpw7c\") pod \"maas-api-e2e-ait-81247cf5-79f5845899-wlv2v\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:00.558686 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.558586    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:00.840850 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.840751    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:00.942452 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.942420    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:00.953305 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.953283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/586eefea-3b96-4f69-ad48-6fc245743509-maas-api-tls\") pod \"maas-api-e2e-ait-81247cf5-79f5845899-wlv2v\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:00.955561 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:00.955538    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/586eefea-3b96-4f69-ad48-6fc245743509-maas-api-tls\") pod \"maas-api-e2e-ait-81247cf5-79f5845899-wlv2v\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:01.164868 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:01.164816    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:01.405363 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:01.405311    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:01.435578 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:01.435553    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"]
Sep 15 12:40:01.437129 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:40:01.437101    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586eefea_3b96_4f69_ad48_6fc245743509.slice/crio-7a82b401d331626fba1dee9f32208ae1e4fe8e5c762b5e6b0ee0cabc39066be5 WatchSource:0}: Error finding container 7a82b401d331626fba1dee9f32208ae1e4fe8e5c762b5e6b0ee0cabc39066be5: Status 404 returned error can't find the container with id 7a82b401d331626fba1dee9f32208ae1e4fe8e5c762b5e6b0ee0cabc39066be5
Sep 15 12:40:02.174056 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:02.174019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v" event={"ID":"586eefea-3b96-4f69-ad48-6fc245743509","Type":"ContainerStarted","Data":"7a82b401d331626fba1dee9f32208ae1e4fe8e5c762b5e6b0ee0cabc39066be5"}
Sep 15 12:40:02.269665 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:02.269620    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:02.850101 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:02.850070    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:03.524671 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:03.524636    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:05.197894 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:05.197862    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v" event={"ID":"586eefea-3b96-4f69-ad48-6fc245743509","Type":"ContainerStarted","Data":"16e3b79e91c706934f7ea2494356c34001d7379ea3a6e7dd94b61406a35fc354"}
Sep 15 12:40:05.198326 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:05.198036    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:05.213680 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:05.213636    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v" podStartSLOduration=1.9080735610000001 podStartE2EDuration="5.213622127s" totalImagesPullingTime="3.305548566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:40:05.213250013 +0000 UTC m=+876.597960888" watchObservedRunningTime="2026-09-15 12:40:05.213622127 +0000 UTC m=+876.598333010"
Sep 15 12:40:08.534983 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.534948    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"]
Sep 15 12:40:08.538372 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.538356    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.540776 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.540750    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-bf37f4\"/\"openshift-service-ca.crt\""
Sep 15 12:40:08.541097 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.541082    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-bf37f4\"/\"kube-root-ca.crt\""
Sep 15 12:40:08.542014 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.541993    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-bf37f4\"/\"autores-model-bf37f4-kserve-self-signed-certs\""
Sep 15 12:40:08.542108 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.542024    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-bf37f4\"/\"default-dockercfg-65cg2\""
Sep 15 12:40:08.547750 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.547723    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"]
Sep 15 12:40:08.624612 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.624560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-tls-certs\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.624612 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.624627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-dshm\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.624911 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.624731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt59w\" (UniqueName: \"kubernetes.io/projected/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-kube-api-access-rt59w\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.624911 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.624773    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-home\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.624911 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.624862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-tmp-dir\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.624911 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.624891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-model-cache\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.725503 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.725466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt59w\" (UniqueName: \"kubernetes.io/projected/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-kube-api-access-rt59w\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.725689 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.725513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-home\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.725689 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.725641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-tmp-dir\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.725689 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.725677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-model-cache\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.725879 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.725720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-tls-certs\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.725879 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.725765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-dshm\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.726001 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.725918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-home\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.726098 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.726075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-tmp-dir\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.726186 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.726165    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-model-cache\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.728014 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.727997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-dshm\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.728308 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.728290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-tls-certs\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.732915 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.732891    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt59w\" (UniqueName: \"kubernetes.io/projected/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-kube-api-access-rt59w\") pod \"autores-model-bf37f4-kserve-676dd9f8c5-8tlqf\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") " pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.848908 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.848878    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:08.871548 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:08.871511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:09.121091 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:09.121066    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"]
Sep 15 12:40:09.121904 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:40:09.121878    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44be1427_7e4b_4c2e_a4b1_28bfcb8e8076.slice/crio-217b5eb21111218124eb0521f4f536b2337acf0eb0bab19b79ce24568fc65c8d WatchSource:0}: Error finding container 217b5eb21111218124eb0521f4f536b2337acf0eb0bab19b79ce24568fc65c8d: Status 404 returned error can't find the container with id 217b5eb21111218124eb0521f4f536b2337acf0eb0bab19b79ce24568fc65c8d
Sep 15 12:40:09.222864 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:09.222669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf" event={"ID":"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076","Type":"ContainerStarted","Data":"02a511ec8959a3e4a7234cf26147c972fa199c184dcdf7f8401b0a70d324a993"}
Sep 15 12:40:09.222864 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:09.222721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf" event={"ID":"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076","Type":"ContainerStarted","Data":"217b5eb21111218124eb0521f4f536b2337acf0eb0bab19b79ce24568fc65c8d"}
Sep 15 12:40:09.245014 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:09.242460    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf" podStartSLOduration=1.242439394 podStartE2EDuration="1.242439394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:40:09.238303659 +0000 UTC m=+880.623014534" watchObservedRunningTime="2026-09-15 12:40:09.242439394 +0000 UTC m=+880.627150270"
Sep 15 12:40:11.208572 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:11.208545    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:13.952236 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:13.952195    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:40:18.849635 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:18.849592    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:18.850164 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:18.849646    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:18.862338 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:18.862303    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:19.291426 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:19.291326    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:40:22.821551 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:22.821453    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:23.271672 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:23.271632    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:24.281125 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:24.281087    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:24.812529 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:24.812499    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:24.913543 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:24.913508    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:25.085265 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:25.085173    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:25.800736 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:25.800701    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:26.909157 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:26.909120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:27.404197 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:27.404160    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:28.164087 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:28.164048    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:28.949915 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:28.949874    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:29.128086 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:29.128062    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:40:29.132519 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:29.132498    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:40:29.725990 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:29.725958    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:30.474109 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.474077    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"]
Sep 15 12:40:30.474357 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.474318    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v" podUID="586eefea-3b96-4f69-ad48-6fc245743509" containerName="maas-api" containerID="cri-o://16e3b79e91c706934f7ea2494356c34001d7379ea3a6e7dd94b61406a35fc354" gracePeriod=30
Sep 15 12:40:30.495949 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.495919    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:30.611701 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.611647    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:30.712102 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.712071    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:30.722216 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.722195    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:30.727518 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.727477    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpw7c\" (UniqueName: \"kubernetes.io/projected/586eefea-3b96-4f69-ad48-6fc245743509-kube-api-access-jpw7c\") pod \"586eefea-3b96-4f69-ad48-6fc245743509\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") "
Sep 15 12:40:30.727518 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.727510    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/586eefea-3b96-4f69-ad48-6fc245743509-maas-api-tls\") pod \"586eefea-3b96-4f69-ad48-6fc245743509\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") "
Sep 15 12:40:30.727944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.727670    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/586eefea-3b96-4f69-ad48-6fc245743509-openshift-service-ca\") pod \"586eefea-3b96-4f69-ad48-6fc245743509\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") "
Sep 15 12:40:30.727944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.727714    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586eefea-3b96-4f69-ad48-6fc245743509-metrics-certs\") pod \"586eefea-3b96-4f69-ad48-6fc245743509\" (UID: \"586eefea-3b96-4f69-ad48-6fc245743509\") "
Sep 15 12:40:30.728043 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.728024    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/586eefea-3b96-4f69-ad48-6fc245743509-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "586eefea-3b96-4f69-ad48-6fc245743509" (UID: "586eefea-3b96-4f69-ad48-6fc245743509"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:40:30.729656 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.729623    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/586eefea-3b96-4f69-ad48-6fc245743509-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "586eefea-3b96-4f69-ad48-6fc245743509" (UID: "586eefea-3b96-4f69-ad48-6fc245743509"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:40:30.729754 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.729649    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586eefea-3b96-4f69-ad48-6fc245743509-kube-api-access-jpw7c" (OuterVolumeSpecName: "kube-api-access-jpw7c") pod "586eefea-3b96-4f69-ad48-6fc245743509" (UID: "586eefea-3b96-4f69-ad48-6fc245743509"). InnerVolumeSpecName "kube-api-access-jpw7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:40:30.729754 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.729727    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/586eefea-3b96-4f69-ad48-6fc245743509-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "586eefea-3b96-4f69-ad48-6fc245743509" (UID: "586eefea-3b96-4f69-ad48-6fc245743509"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:40:30.792265 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.792228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:30.828500 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.828473    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/586eefea-3b96-4f69-ad48-6fc245743509-openshift-service-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:40:30.828500 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.828498    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/586eefea-3b96-4f69-ad48-6fc245743509-metrics-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:40:30.828671 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.828508    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpw7c\" (UniqueName: \"kubernetes.io/projected/586eefea-3b96-4f69-ad48-6fc245743509-kube-api-access-jpw7c\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:40:30.828671 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.828517    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/586eefea-3b96-4f69-ad48-6fc245743509-maas-api-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:40:30.851402 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:30.851366    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:31.342568 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:31.342538    2579 generic.go:356] "Generic (PLEG): container finished" podID="586eefea-3b96-4f69-ad48-6fc245743509" containerID="16e3b79e91c706934f7ea2494356c34001d7379ea3a6e7dd94b61406a35fc354" exitCode=0
Sep 15 12:40:31.342736 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:31.342578    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"
Sep 15 12:40:31.342736 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:31.342636    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v" event={"ID":"586eefea-3b96-4f69-ad48-6fc245743509","Type":"ContainerDied","Data":"16e3b79e91c706934f7ea2494356c34001d7379ea3a6e7dd94b61406a35fc354"}
Sep 15 12:40:31.342736 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:31.342684    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v" event={"ID":"586eefea-3b96-4f69-ad48-6fc245743509","Type":"ContainerDied","Data":"7a82b401d331626fba1dee9f32208ae1e4fe8e5c762b5e6b0ee0cabc39066be5"}
Sep 15 12:40:31.342736 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:31.342706    2579 scope.go:117] "RemoveContainer" containerID="16e3b79e91c706934f7ea2494356c34001d7379ea3a6e7dd94b61406a35fc354"
Sep 15 12:40:31.351726 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:31.351707    2579 scope.go:117] "RemoveContainer" containerID="16e3b79e91c706934f7ea2494356c34001d7379ea3a6e7dd94b61406a35fc354"
Sep 15 12:40:31.352020 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:40:31.351998    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e3b79e91c706934f7ea2494356c34001d7379ea3a6e7dd94b61406a35fc354\": container with ID starting with 16e3b79e91c706934f7ea2494356c34001d7379ea3a6e7dd94b61406a35fc354 not found: ID does not exist" containerID="16e3b79e91c706934f7ea2494356c34001d7379ea3a6e7dd94b61406a35fc354"
Sep 15 12:40:31.352100 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:31.352033    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e3b79e91c706934f7ea2494356c34001d7379ea3a6e7dd94b61406a35fc354"} err="failed to get container status \"16e3b79e91c706934f7ea2494356c34001d7379ea3a6e7dd94b61406a35fc354\": rpc error: code = NotFound desc = could not find container \"16e3b79e91c706934f7ea2494356c34001d7379ea3a6e7dd94b61406a35fc354\": container with ID starting with 16e3b79e91c706934f7ea2494356c34001d7379ea3a6e7dd94b61406a35fc354 not found: ID does not exist"
Sep 15 12:40:31.359066 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:31.359040    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"]
Sep 15 12:40:31.365179 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:31.365127    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-81247cf5-79f5845899-wlv2v"]
Sep 15 12:40:31.773865 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:31.773764    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:31.899391 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:31.899360    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:31.991299 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:31.991266    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:32.312497 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:32.312462    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:32.607098 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:32.607062    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:32.874938 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:32.874817    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:33.105555 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:33.105523    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="586eefea-3b96-4f69-ad48-6fc245743509" path="/var/lib/kubelet/pods/586eefea-3b96-4f69-ad48-6fc245743509/volumes"
Sep 15 12:40:33.105555 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:33.105543    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:33.620342 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:33.620303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:34.391614 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:34.391570    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:35.574688 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:35.574657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:35.674400 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:35.674364    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:35.926653 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:35.926615    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:36.708416 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:36.708381    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:37.585200 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:37.585159    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:38.261335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:38.261301    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:38.913817 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:38.913781    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:39.760325 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.760294    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"]
Sep 15 12:40:39.762681 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.760965    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="586eefea-3b96-4f69-ad48-6fc245743509" containerName="maas-api"
Sep 15 12:40:39.762681 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.760985    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="586eefea-3b96-4f69-ad48-6fc245743509" containerName="maas-api"
Sep 15 12:40:39.762681 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.761068    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="586eefea-3b96-4f69-ad48-6fc245743509" containerName="maas-api"
Sep 15 12:40:39.764275 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.764238    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:39.768053 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.768025    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1459b7a2-serving-cert\""
Sep 15 12:40:39.778210 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.778064    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"]
Sep 15 12:40:39.796160 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.796129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:39.808406 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.808376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbbaca3e-6a5e-4496-ab19-092d22755b78-maas-api-tls\") pod \"maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:39.808527 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.808427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbaca3e-6a5e-4496-ab19-092d22755b78-openshift-service-ca\") pod \"maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:39.808527 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.808469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbbaca3e-6a5e-4496-ab19-092d22755b78-metrics-certs\") pod \"maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:39.808527 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.808503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22wjg\" (UniqueName: \"kubernetes.io/projected/fbbaca3e-6a5e-4496-ab19-092d22755b78-kube-api-access-22wjg\") pod \"maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:39.897195 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.897162    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:39.910141 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.910112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbbaca3e-6a5e-4496-ab19-092d22755b78-maas-api-tls\") pod \"maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:39.910261 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.910160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbaca3e-6a5e-4496-ab19-092d22755b78-openshift-service-ca\") pod \"maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:39.910261 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.910205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbbaca3e-6a5e-4496-ab19-092d22755b78-metrics-certs\") pod \"maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:39.910261 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.910241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22wjg\" (UniqueName: \"kubernetes.io/projected/fbbaca3e-6a5e-4496-ab19-092d22755b78-kube-api-access-22wjg\") pod \"maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:39.910382 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:40:39.910259    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-serving-cert: secret "maas-api-e2e-derive-1459b7a2-serving-cert" not found
Sep 15 12:40:39.910382 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:40:39.910341    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbbaca3e-6a5e-4496-ab19-092d22755b78-maas-api-tls podName:fbbaca3e-6a5e-4496-ab19-092d22755b78 nodeName:}" failed. No retries permitted until 2026-09-15 12:40:40.410319321 +0000 UTC m=+911.795030197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbbaca3e-6a5e-4496-ab19-092d22755b78-maas-api-tls") pod "maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht" (UID: "fbbaca3e-6a5e-4496-ab19-092d22755b78") : secret "maas-api-e2e-derive-1459b7a2-serving-cert" not found
Sep 15 12:40:39.910788 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.910770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbaca3e-6a5e-4496-ab19-092d22755b78-openshift-service-ca\") pod \"maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:39.912713 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.912689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbbaca3e-6a5e-4496-ab19-092d22755b78-metrics-certs\") pod \"maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:39.918641 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.918615    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22wjg\" (UniqueName: \"kubernetes.io/projected/fbbaca3e-6a5e-4496-ab19-092d22755b78-kube-api-access-22wjg\") pod \"maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:39.996937 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:39.996901    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:40.262921 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:40.262885    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:40.281788 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:40.281756    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:40:40.359233 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:40.359192    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:40.416185 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:40.416151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbbaca3e-6a5e-4496-ab19-092d22755b78-maas-api-tls\") pod \"maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:40.418857 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:40.418814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbbaca3e-6a5e-4496-ab19-092d22755b78-maas-api-tls\") pod \"maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:40.686903 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:40.686862    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:40.957913 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:40.957884    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"]
Sep 15 12:40:40.959849 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:40:40.959806    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbaca3e_6a5e_4496_ab19_092d22755b78.slice/crio-89f416c66606d7793448aaa657e10a48c160999aff546d321b5f476e89d829c7 WatchSource:0}: Error finding container 89f416c66606d7793448aaa657e10a48c160999aff546d321b5f476e89d829c7: Status 404 returned error can't find the container with id 89f416c66606d7793448aaa657e10a48c160999aff546d321b5f476e89d829c7
Sep 15 12:40:41.396061 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:41.396028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht" event={"ID":"fbbaca3e-6a5e-4496-ab19-092d22755b78","Type":"ContainerStarted","Data":"89f416c66606d7793448aaa657e10a48c160999aff546d321b5f476e89d829c7"}
Sep 15 12:40:42.403964 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:42.403930    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht" event={"ID":"fbbaca3e-6a5e-4496-ab19-092d22755b78","Type":"ContainerStarted","Data":"1a02c23d8cb0df4520105a385dd6c9ac062e56f9ae327978ae70b2305e052665"}
Sep 15 12:40:42.404381 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:42.404028    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:42.421537 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:42.421476    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht" podStartSLOduration=2.654913793 podStartE2EDuration="3.421453968s" totalImagesPullingTime="766.540175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:40:42.42069956 +0000 UTC m=+913.805410453" watchObservedRunningTime="2026-09-15 12:40:42.421453968 +0000 UTC m=+913.806164860"
Sep 15 12:40:44.508778 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:44.508746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:45.485909 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:45.485859    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:46.132096 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:46.132066    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:46.891066 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:46.891029    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:47.680408 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:47.680368    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:48.362213 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:48.362177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:48.415770 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:48.415742    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:40:49.102736 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:49.102701    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:49.768539 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:49.768506    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:53.078307 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:53.078278    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:54.870644 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:54.870611    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"]
Sep 15 12:40:54.871074 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:54.870864    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c" podUID="40838255-e5ea-4358-8f95-64cc84217c5c" containerName="maas-api" containerID="cri-o://b7f4b90525e89fa184a88e59f6a41896d6246116145316c3e9979357cf4674b3" gracePeriod=30
Sep 15 12:40:54.894192 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:54.894164    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:54.994953 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:54.994921    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:55.095427 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.095394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:55.118995 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.118971    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:40:55.195097 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.195020    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:55.257669 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.257636    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40838255-e5ea-4358-8f95-64cc84217c5c-maas-api-tls\") pod \"40838255-e5ea-4358-8f95-64cc84217c5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") "
Sep 15 12:40:55.257820 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.257687    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbkgh\" (UniqueName: \"kubernetes.io/projected/40838255-e5ea-4358-8f95-64cc84217c5c-kube-api-access-tbkgh\") pod \"40838255-e5ea-4358-8f95-64cc84217c5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") "
Sep 15 12:40:55.257820 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.257760    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40838255-e5ea-4358-8f95-64cc84217c5c-metrics-certs\") pod \"40838255-e5ea-4358-8f95-64cc84217c5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") "
Sep 15 12:40:55.257985 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.257861    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40838255-e5ea-4358-8f95-64cc84217c5c-openshift-service-ca\") pod \"40838255-e5ea-4358-8f95-64cc84217c5c\" (UID: \"40838255-e5ea-4358-8f95-64cc84217c5c\") "
Sep 15 12:40:55.258257 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.258233    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40838255-e5ea-4358-8f95-64cc84217c5c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40838255-e5ea-4358-8f95-64cc84217c5c" (UID: "40838255-e5ea-4358-8f95-64cc84217c5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:40:55.260168 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.260128    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40838255-e5ea-4358-8f95-64cc84217c5c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40838255-e5ea-4358-8f95-64cc84217c5c" (UID: "40838255-e5ea-4358-8f95-64cc84217c5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:40:55.260442 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.260409    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40838255-e5ea-4358-8f95-64cc84217c5c-kube-api-access-tbkgh" (OuterVolumeSpecName: "kube-api-access-tbkgh") pod "40838255-e5ea-4358-8f95-64cc84217c5c" (UID: "40838255-e5ea-4358-8f95-64cc84217c5c"). InnerVolumeSpecName "kube-api-access-tbkgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:40:55.260562 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.260439    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40838255-e5ea-4358-8f95-64cc84217c5c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "40838255-e5ea-4358-8f95-64cc84217c5c" (UID: "40838255-e5ea-4358-8f95-64cc84217c5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:40:55.358778 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.358743    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40838255-e5ea-4358-8f95-64cc84217c5c-maas-api-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:40:55.358778 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.358774    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbkgh\" (UniqueName: \"kubernetes.io/projected/40838255-e5ea-4358-8f95-64cc84217c5c-kube-api-access-tbkgh\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:40:55.358778 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.358784    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40838255-e5ea-4358-8f95-64cc84217c5c-metrics-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:40:55.359029 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.358793    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40838255-e5ea-4358-8f95-64cc84217c5c-openshift-service-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:40:55.485671 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.484335    2579 generic.go:356] "Generic (PLEG): container finished" podID="40838255-e5ea-4358-8f95-64cc84217c5c" containerID="b7f4b90525e89fa184a88e59f6a41896d6246116145316c3e9979357cf4674b3" exitCode=0
Sep 15 12:40:55.485671 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.484533    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c" event={"ID":"40838255-e5ea-4358-8f95-64cc84217c5c","Type":"ContainerDied","Data":"b7f4b90525e89fa184a88e59f6a41896d6246116145316c3e9979357cf4674b3"}
Sep 15 12:40:55.485671 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.484623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c" event={"ID":"40838255-e5ea-4358-8f95-64cc84217c5c","Type":"ContainerDied","Data":"95ffa4e72bc096d2b7eb0d4e15b003156bb3dff288b5a4c6dd816d8ceeb9755e"}
Sep 15 12:40:55.485671 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.484663    2579 scope.go:117] "RemoveContainer" containerID="b7f4b90525e89fa184a88e59f6a41896d6246116145316c3e9979357cf4674b3"
Sep 15 12:40:55.485671 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.484892    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"
Sep 15 12:40:55.491339 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.491273    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:55.496148 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.496130    2579 scope.go:117] "RemoveContainer" containerID="b7f4b90525e89fa184a88e59f6a41896d6246116145316c3e9979357cf4674b3"
Sep 15 12:40:55.496407 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:40:55.496390    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f4b90525e89fa184a88e59f6a41896d6246116145316c3e9979357cf4674b3\": container with ID starting with b7f4b90525e89fa184a88e59f6a41896d6246116145316c3e9979357cf4674b3 not found: ID does not exist" containerID="b7f4b90525e89fa184a88e59f6a41896d6246116145316c3e9979357cf4674b3"
Sep 15 12:40:55.496452 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.496413    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f4b90525e89fa184a88e59f6a41896d6246116145316c3e9979357cf4674b3"} err="failed to get container status \"b7f4b90525e89fa184a88e59f6a41896d6246116145316c3e9979357cf4674b3\": rpc error: code = NotFound desc = could not find container \"b7f4b90525e89fa184a88e59f6a41896d6246116145316c3e9979357cf4674b3\": container with ID starting with b7f4b90525e89fa184a88e59f6a41896d6246116145316c3e9979357cf4674b3 not found: ID does not exist"
Sep 15 12:40:55.507638 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.507613    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"]
Sep 15 12:40:55.512032 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.512012    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-bf37f4-5c876cb6fb-9kp5c"]
Sep 15 12:40:55.693791 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:55.693755    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:56.504852 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:56.504792    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:57.105555 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:57.105518    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40838255-e5ea-4358-8f95-64cc84217c5c" path="/var/lib/kubelet/pods/40838255-e5ea-4358-8f95-64cc84217c5c/volumes"
Sep 15 12:40:57.580676 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:57.580585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:58.410948 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:58.410912    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:59.318493 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:59.318457    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:40:59.980348 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:40:59.980312    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:00.471900 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:00.471863    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"]
Sep 15 12:41:00.568493 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:00.568464    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:00.886047 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:00.886007    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:01.710470 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:01.710434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:01.956937 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:01.956900    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:41:02.384914 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:02.384883    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:03.180008 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:03.179979    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:03.879910 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:03.879877    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:04.704448 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:04.704413    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:06.661384 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.661354    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"]
Sep 15 12:41:06.661771 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.661709    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40838255-e5ea-4358-8f95-64cc84217c5c" containerName="maas-api"
Sep 15 12:41:06.661771 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.661721    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="40838255-e5ea-4358-8f95-64cc84217c5c" containerName="maas-api"
Sep 15 12:41:06.661869 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.661782    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="40838255-e5ea-4358-8f95-64cc84217c5c" containerName="maas-api"
Sep 15 12:41:06.665088 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.665066    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.667664 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.667639    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4609ba\"/\"kube-root-ca.crt\""
Sep 15 12:41:06.668598 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.668581    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4609ba\"/\"default-dockercfg-2zpkj\""
Sep 15 12:41:06.668731 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.668719    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4609ba\"/\"openshift-service-ca.crt\""
Sep 15 12:41:06.668792 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.668775    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4609ba\"/\"test-model-4609ba-kserve-self-signed-certs\""
Sep 15 12:41:06.673002 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.672664    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"]
Sep 15 12:41:06.690559 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.690523    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:06.763663 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.763629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-home\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.763842 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.763687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-model-cache\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.763842 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.763720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-tmp-dir\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.763842 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.763739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-tls-certs\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.763842 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.763757    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-dshm\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.763842 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.763789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtcvm\" (UniqueName: \"kubernetes.io/projected/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-kube-api-access-rtcvm\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.865146 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.865107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-model-cache\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.865146 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.865153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-tmp-dir\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.865389 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.865171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-tls-certs\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.865389 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.865191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-dshm\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.865389 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.865215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtcvm\" (UniqueName: \"kubernetes.io/projected/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-kube-api-access-rtcvm\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.865389 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.865310    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-home\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.865624 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.865599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-model-cache\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.865696 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.865612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-tmp-dir\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.865696 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.865679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-home\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.867586 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.867551    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-dshm\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.867744 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.867725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-tls-certs\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.874097 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.874073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtcvm\" (UniqueName: \"kubernetes.io/projected/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-kube-api-access-rtcvm\") pod \"test-model-4609ba-kserve-649698b8d9-44j76\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") " pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:06.908784 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.908747    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:06.975476 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:06.975388    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:07.255149 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.255121    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"]
Sep 15 12:41:07.257402 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:41:07.257377    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e232a06_87d3_4b04_9ff1_afbe6b0cf4c3.slice/crio-f0df293d56d732763f9ea5c66b612dc5d7f974a51e89fc8f1621da47a1ac11fe WatchSource:0}: Error finding container f0df293d56d732763f9ea5c66b612dc5d7f974a51e89fc8f1621da47a1ac11fe: Status 404 returned error can't find the container with id f0df293d56d732763f9ea5c66b612dc5d7f974a51e89fc8f1621da47a1ac11fe
Sep 15 12:41:07.548939 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.548821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76" event={"ID":"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3","Type":"ContainerStarted","Data":"644873ecee75a0a02b676a6b1d3bfd99b9c6199e62181d8d7069bf71b2bacb47"}
Sep 15 12:41:07.548939 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.548890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76" event={"ID":"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3","Type":"ContainerStarted","Data":"f0df293d56d732763f9ea5c66b612dc5d7f974a51e89fc8f1621da47a1ac11fe"}
Sep 15 12:41:07.570593 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.570540    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76" podStartSLOduration=1.570522636 podStartE2EDuration="1.570522636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:41:07.569614642 +0000 UTC m=+938.954325527" watchObservedRunningTime="2026-09-15 12:41:07.570522636 +0000 UTC m=+938.955233519"
Sep 15 12:41:07.620762 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.620725    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"]
Sep 15 12:41:07.621034 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.621001    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht" podUID="fbbaca3e-6a5e-4496-ab19-092d22755b78" containerName="maas-api" containerID="cri-o://1a02c23d8cb0df4520105a385dd6c9ac062e56f9ae327978ae70b2305e052665" gracePeriod=30
Sep 15 12:41:07.651339 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.651308    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:07.718054 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.718018    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:07.776362 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.776325    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:07.856965 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.856941    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:41:07.878460 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.878423    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:07.974290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.974263    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbbaca3e-6a5e-4496-ab19-092d22755b78-maas-api-tls\") pod \"fbbaca3e-6a5e-4496-ab19-092d22755b78\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") "
Sep 15 12:41:07.974448 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.974301    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22wjg\" (UniqueName: \"kubernetes.io/projected/fbbaca3e-6a5e-4496-ab19-092d22755b78-kube-api-access-22wjg\") pod \"fbbaca3e-6a5e-4496-ab19-092d22755b78\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") "
Sep 15 12:41:07.974448 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.974335    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbbaca3e-6a5e-4496-ab19-092d22755b78-metrics-certs\") pod \"fbbaca3e-6a5e-4496-ab19-092d22755b78\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") "
Sep 15 12:41:07.974448 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.974379    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbaca3e-6a5e-4496-ab19-092d22755b78-openshift-service-ca\") pod \"fbbaca3e-6a5e-4496-ab19-092d22755b78\" (UID: \"fbbaca3e-6a5e-4496-ab19-092d22755b78\") "
Sep 15 12:41:07.974850 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.974808    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbaca3e-6a5e-4496-ab19-092d22755b78-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fbbaca3e-6a5e-4496-ab19-092d22755b78" (UID: "fbbaca3e-6a5e-4496-ab19-092d22755b78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:41:07.976625 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.976598    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbaca3e-6a5e-4496-ab19-092d22755b78-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fbbaca3e-6a5e-4496-ab19-092d22755b78" (UID: "fbbaca3e-6a5e-4496-ab19-092d22755b78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:41:07.976968 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.976945    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbaca3e-6a5e-4496-ab19-092d22755b78-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fbbaca3e-6a5e-4496-ab19-092d22755b78" (UID: "fbbaca3e-6a5e-4496-ab19-092d22755b78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:41:07.977065 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.976949    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbaca3e-6a5e-4496-ab19-092d22755b78-kube-api-access-22wjg" (OuterVolumeSpecName: "kube-api-access-22wjg") pod "fbbaca3e-6a5e-4496-ab19-092d22755b78" (UID: "fbbaca3e-6a5e-4496-ab19-092d22755b78"). InnerVolumeSpecName "kube-api-access-22wjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:41:07.978214 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:07.978191    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:08.058313 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.058284    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:08.075446 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.075419    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbbaca3e-6a5e-4496-ab19-092d22755b78-metrics-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:41:08.075446 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.075445    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbaca3e-6a5e-4496-ab19-092d22755b78-openshift-service-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:41:08.075651 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.075456    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbbaca3e-6a5e-4496-ab19-092d22755b78-maas-api-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:41:08.075651 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.075468    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22wjg\" (UniqueName: \"kubernetes.io/projected/fbbaca3e-6a5e-4496-ab19-092d22755b78-kube-api-access-22wjg\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:41:08.424934 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.424902    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:08.556483 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.556451    2579 generic.go:356] "Generic (PLEG): container finished" podID="fbbaca3e-6a5e-4496-ab19-092d22755b78" containerID="1a02c23d8cb0df4520105a385dd6c9ac062e56f9ae327978ae70b2305e052665" exitCode=0
Sep 15 12:41:08.556662 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.556501    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"
Sep 15 12:41:08.556662 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.556554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht" event={"ID":"fbbaca3e-6a5e-4496-ab19-092d22755b78","Type":"ContainerDied","Data":"1a02c23d8cb0df4520105a385dd6c9ac062e56f9ae327978ae70b2305e052665"}
Sep 15 12:41:08.556662 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.556623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht" event={"ID":"fbbaca3e-6a5e-4496-ab19-092d22755b78","Type":"ContainerDied","Data":"89f416c66606d7793448aaa657e10a48c160999aff546d321b5f476e89d829c7"}
Sep 15 12:41:08.556662 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.556658    2579 scope.go:117] "RemoveContainer" containerID="1a02c23d8cb0df4520105a385dd6c9ac062e56f9ae327978ae70b2305e052665"
Sep 15 12:41:08.566750 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.566731    2579 scope.go:117] "RemoveContainer" containerID="1a02c23d8cb0df4520105a385dd6c9ac062e56f9ae327978ae70b2305e052665"
Sep 15 12:41:08.567061 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:41:08.567043    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a02c23d8cb0df4520105a385dd6c9ac062e56f9ae327978ae70b2305e052665\": container with ID starting with 1a02c23d8cb0df4520105a385dd6c9ac062e56f9ae327978ae70b2305e052665 not found: ID does not exist" containerID="1a02c23d8cb0df4520105a385dd6c9ac062e56f9ae327978ae70b2305e052665"
Sep 15 12:41:08.567118 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.567070    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a02c23d8cb0df4520105a385dd6c9ac062e56f9ae327978ae70b2305e052665"} err="failed to get container status \"1a02c23d8cb0df4520105a385dd6c9ac062e56f9ae327978ae70b2305e052665\": rpc error: code = NotFound desc = could not find container \"1a02c23d8cb0df4520105a385dd6c9ac062e56f9ae327978ae70b2305e052665\": container with ID starting with 1a02c23d8cb0df4520105a385dd6c9ac062e56f9ae327978ae70b2305e052665 not found: ID does not exist"
Sep 15 12:41:08.581195 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.581171    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"]
Sep 15 12:41:08.583344 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.583325    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1459b7a2-5947bdcf6f-4ddht"]
Sep 15 12:41:08.677247 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.677137    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:08.774448 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.774416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:08.874716 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:08.874677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:09.105814 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:09.105783    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbbaca3e-6a5e-4496-ab19-092d22755b78" path="/var/lib/kubelet/pods/fbbaca3e-6a5e-4496-ab19-092d22755b78/volumes"
Sep 15 12:41:09.398501 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:09.398401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:10.205448 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:10.205404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:10.391925 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:10.391891    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:11.083527 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:11.083492    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:11.872414 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:11.872377    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:15.348592 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.348549    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:15.490767 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.490733    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf" podUID="44be1427-7e4b-4c2e-a4b1-28bfcb8e8076" containerName="main" containerID="cri-o://02a511ec8959a3e4a7234cf26147c972fa199c184dcdf7f8401b0a70d324a993" gracePeriod=45
Sep 15 12:41:15.735212 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.735187    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:41:15.844953 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.844918    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-dshm\") pod \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") "
Sep 15 12:41:15.845143 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.844982    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-model-cache\") pod \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") "
Sep 15 12:41:15.845143 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.845027    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-home\") pod \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") "
Sep 15 12:41:15.845143 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.845046    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-tmp-dir\") pod \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") "
Sep 15 12:41:15.845143 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.845091    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt59w\" (UniqueName: \"kubernetes.io/projected/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-kube-api-access-rt59w\") pod \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") "
Sep 15 12:41:15.845143 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.845115    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-tls-certs\") pod \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\" (UID: \"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076\") "
Sep 15 12:41:15.845419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.845245    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-model-cache" (OuterVolumeSpecName: "model-cache") pod "44be1427-7e4b-4c2e-a4b1-28bfcb8e8076" (UID: "44be1427-7e4b-4c2e-a4b1-28bfcb8e8076"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:41:15.845419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.845264    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-home" (OuterVolumeSpecName: "home") pod "44be1427-7e4b-4c2e-a4b1-28bfcb8e8076" (UID: "44be1427-7e4b-4c2e-a4b1-28bfcb8e8076"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:41:15.845419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.845362    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44be1427-7e4b-4c2e-a4b1-28bfcb8e8076" (UID: "44be1427-7e4b-4c2e-a4b1-28bfcb8e8076"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:41:15.845419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.845389    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:41:15.845419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.845411    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:41:15.847232 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.847201    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-kube-api-access-rt59w" (OuterVolumeSpecName: "kube-api-access-rt59w") pod "44be1427-7e4b-4c2e-a4b1-28bfcb8e8076" (UID: "44be1427-7e4b-4c2e-a4b1-28bfcb8e8076"). InnerVolumeSpecName "kube-api-access-rt59w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:41:15.847321 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.847247    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-dshm" (OuterVolumeSpecName: "dshm") pod "44be1427-7e4b-4c2e-a4b1-28bfcb8e8076" (UID: "44be1427-7e4b-4c2e-a4b1-28bfcb8e8076"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:41:15.847321 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.847248    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44be1427-7e4b-4c2e-a4b1-28bfcb8e8076" (UID: "44be1427-7e4b-4c2e-a4b1-28bfcb8e8076"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:41:15.952215 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.946776    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:41:15.952215 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.946815    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt59w\" (UniqueName: \"kubernetes.io/projected/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-kube-api-access-rt59w\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:41:15.952215 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.946860    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:41:15.952215 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:15.946875    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:41:16.601979 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:16.601943    2579 generic.go:356] "Generic (PLEG): container finished" podID="44be1427-7e4b-4c2e-a4b1-28bfcb8e8076" containerID="02a511ec8959a3e4a7234cf26147c972fa199c184dcdf7f8401b0a70d324a993" exitCode=0
Sep 15 12:41:16.602438 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:16.602004    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"
Sep 15 12:41:16.602438 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:16.602024    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf" event={"ID":"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076","Type":"ContainerDied","Data":"02a511ec8959a3e4a7234cf26147c972fa199c184dcdf7f8401b0a70d324a993"}
Sep 15 12:41:16.602438 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:16.602071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf" event={"ID":"44be1427-7e4b-4c2e-a4b1-28bfcb8e8076","Type":"ContainerDied","Data":"217b5eb21111218124eb0521f4f536b2337acf0eb0bab19b79ce24568fc65c8d"}
Sep 15 12:41:16.602438 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:16.602093    2579 scope.go:117] "RemoveContainer" containerID="02a511ec8959a3e4a7234cf26147c972fa199c184dcdf7f8401b0a70d324a993"
Sep 15 12:41:16.611574 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:16.611550    2579 scope.go:117] "RemoveContainer" containerID="02a511ec8959a3e4a7234cf26147c972fa199c184dcdf7f8401b0a70d324a993"
Sep 15 12:41:16.611954 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:41:16.611931    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a511ec8959a3e4a7234cf26147c972fa199c184dcdf7f8401b0a70d324a993\": container with ID starting with 02a511ec8959a3e4a7234cf26147c972fa199c184dcdf7f8401b0a70d324a993 not found: ID does not exist" containerID="02a511ec8959a3e4a7234cf26147c972fa199c184dcdf7f8401b0a70d324a993"
Sep 15 12:41:16.612028 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:16.611961    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a511ec8959a3e4a7234cf26147c972fa199c184dcdf7f8401b0a70d324a993"} err="failed to get container status \"02a511ec8959a3e4a7234cf26147c972fa199c184dcdf7f8401b0a70d324a993\": rpc error: code = NotFound desc = could not find container \"02a511ec8959a3e4a7234cf26147c972fa199c184dcdf7f8401b0a70d324a993\": container with ID starting with 02a511ec8959a3e4a7234cf26147c972fa199c184dcdf7f8401b0a70d324a993 not found: ID does not exist"
Sep 15 12:41:16.624751 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:16.624717    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"]
Sep 15 12:41:16.632204 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:16.632171    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-bf37f4/autores-model-bf37f4-kserve-676dd9f8c5-8tlqf"]
Sep 15 12:41:16.976236 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:16.976133    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:16.976236 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:16.976212    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:16.989005 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:16.988972    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:17.108594 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:17.108549    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44be1427-7e4b-4c2e-a4b1-28bfcb8e8076" path="/var/lib/kubelet/pods/44be1427-7e4b-4c2e-a4b1-28bfcb8e8076/volumes"
Sep 15 12:41:17.621212 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:17.621180    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:41:18.240481 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:18.240445    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:18.341694 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:18.341659    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:26.356056 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:26.356023    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:41:33.697695 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:33.697655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:33.941044 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:33.941011    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:34.385298 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:34.385265    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:34.695035 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:34.694950    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:34.842419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:34.842386    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:36.782528 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:36.782493    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:37.452349 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:37.452315    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:37.554744 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:37.554704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:40.425744 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.425709    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:40.671153 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.671117    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"]
Sep 15 12:41:40.671459 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.671447    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44be1427-7e4b-4c2e-a4b1-28bfcb8e8076" containerName="main"
Sep 15 12:41:40.671511 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.671461    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="44be1427-7e4b-4c2e-a4b1-28bfcb8e8076" containerName="main"
Sep 15 12:41:40.671511 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.671486    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbbaca3e-6a5e-4496-ab19-092d22755b78" containerName="maas-api"
Sep 15 12:41:40.671511 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.671492    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbbaca3e-6a5e-4496-ab19-092d22755b78" containerName="maas-api"
Sep 15 12:41:40.671618 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.671550    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbbaca3e-6a5e-4496-ab19-092d22755b78" containerName="maas-api"
Sep 15 12:41:40.671618 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.671559    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="44be1427-7e4b-4c2e-a4b1-28bfcb8e8076" containerName="main"
Sep 15 12:41:40.675948 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.675888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:40.678408 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.678387    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-0f955e-serving-cert\""
Sep 15 12:41:40.687184 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.687157    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"]
Sep 15 12:41:40.754490 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.754456    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:40.779568 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.779542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5t7\" (UniqueName: \"kubernetes.io/projected/c670ef38-d3c6-48e5-be9d-c9734b0b979c-kube-api-access-lh5t7\") pod \"maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:40.779708 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.779578    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-metrics-certs\") pod \"maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:40.779708 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.779660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c670ef38-d3c6-48e5-be9d-c9734b0b979c-openshift-service-ca\") pod \"maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:40.779784 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.779721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-maas-api-tls\") pod \"maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:40.854827 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.854801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:40.880578 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.880554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c670ef38-d3c6-48e5-be9d-c9734b0b979c-openshift-service-ca\") pod \"maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:40.880738 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.880605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-maas-api-tls\") pod \"maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:40.880738 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.880643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5t7\" (UniqueName: \"kubernetes.io/projected/c670ef38-d3c6-48e5-be9d-c9734b0b979c-kube-api-access-lh5t7\") pod \"maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:40.880738 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.880661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-metrics-certs\") pod \"maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:40.880945 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:41:40.880775    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-serving-cert: secret "maas-api-e2e-explicit-0f955e-serving-cert" not found
Sep 15 12:41:40.880945 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:41:40.880867    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-maas-api-tls podName:c670ef38-d3c6-48e5-be9d-c9734b0b979c nodeName:}" failed. No retries permitted until 2026-09-15 12:41:41.380824457 +0000 UTC m=+972.765535340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-maas-api-tls") pod "maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2" (UID: "c670ef38-d3c6-48e5-be9d-c9734b0b979c") : secret "maas-api-e2e-explicit-0f955e-serving-cert" not found
Sep 15 12:41:40.881329 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.881310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c670ef38-d3c6-48e5-be9d-c9734b0b979c-openshift-service-ca\") pod \"maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:40.883163 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.883138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-metrics-certs\") pod \"maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:40.892187 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:40.892166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5t7\" (UniqueName: \"kubernetes.io/projected/c670ef38-d3c6-48e5-be9d-c9734b0b979c-kube-api-access-lh5t7\") pod \"maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:41.164161 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:41.164129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:41.262010 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:41.261980    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:41.385686 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:41.385657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-maas-api-tls\") pod \"maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:41.385861 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:41:41.385815    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-serving-cert: secret "maas-api-e2e-explicit-0f955e-serving-cert" not found
Sep 15 12:41:41.385924 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:41:41.385913    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-maas-api-tls podName:c670ef38-d3c6-48e5-be9d-c9734b0b979c nodeName:}" failed. No retries permitted until 2026-09-15 12:41:42.385895067 +0000 UTC m=+973.770605934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-maas-api-tls") pod "maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2" (UID: "c670ef38-d3c6-48e5-be9d-c9734b0b979c") : secret "maas-api-e2e-explicit-0f955e-serving-cert" not found
Sep 15 12:41:41.622268 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:41.622234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:42.393617 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:42.393582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-maas-api-tls\") pod \"maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:42.396010 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:42.395981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-maas-api-tls\") pod \"maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:42.487880 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:42.487818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:42.665028 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:42.664935    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:42.757711 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:42.757689    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"]
Sep 15 12:41:42.759577 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:41:42.759544    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc670ef38_d3c6_48e5_be9d_c9734b0b979c.slice/crio-608ac513a65b64b0c674ec4942682aa32c41f9bbae8ca33d3fd0a1ad3db7eb21 WatchSource:0}: Error finding container 608ac513a65b64b0c674ec4942682aa32c41f9bbae8ca33d3fd0a1ad3db7eb21: Status 404 returned error can't find the container with id 608ac513a65b64b0c674ec4942682aa32c41f9bbae8ca33d3fd0a1ad3db7eb21
Sep 15 12:41:42.760921 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:42.760897    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:41:43.755225 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:43.755188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2" event={"ID":"c670ef38-d3c6-48e5-be9d-c9734b0b979c","Type":"ContainerStarted","Data":"4e16caf80dcb9e882cf011b12757ff3027ecd99a8f99babbad6e25dcad1e30ce"}
Sep 15 12:41:43.755619 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:43.755248    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2" event={"ID":"c670ef38-d3c6-48e5-be9d-c9734b0b979c","Type":"ContainerStarted","Data":"608ac513a65b64b0c674ec4942682aa32c41f9bbae8ca33d3fd0a1ad3db7eb21"}
Sep 15 12:41:43.755619 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:43.755288    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:43.771664 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:43.771550    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2" podStartSLOduration=3.414001496 podStartE2EDuration="3.771530053s" totalImagesPullingTime="357.528557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:41:43.771099774 +0000 UTC m=+975.155810658" watchObservedRunningTime="2026-09-15 12:41:43.771530053 +0000 UTC m=+975.156240938"
Sep 15 12:41:45.837440 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:45.837406    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"]
Sep 15 12:41:45.840786 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:45.840770    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:45.843225 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:45.843198    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ee96ec\"/\"premium-w4-ee96ec-kserve-self-signed-certs\""
Sep 15 12:41:45.851077 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:45.851055    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"]
Sep 15 12:41:45.927815 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:45.927783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-home\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:45.927815 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:45.927819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-dshm\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:45.928038 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:45.927899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-tmp-dir\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:45.928038 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:45.927941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-tls-certs\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:45.928038 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:45.927969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-model-cache\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:45.928038 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:45.927990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bst2d\" (UniqueName: \"kubernetes.io/projected/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-kube-api-access-bst2d\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:46.028422 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.028389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-home\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:46.028422 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.028424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-dshm\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:46.028657 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.028461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-tmp-dir\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:46.028657 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.028496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-tls-certs\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:46.028657 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.028533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-model-cache\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:46.028657 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.028558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bst2d\" (UniqueName: \"kubernetes.io/projected/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-kube-api-access-bst2d\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:46.028911 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.028850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-home\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:46.028964 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.028900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-tmp-dir\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:46.029139 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.029112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-model-cache\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:46.030900 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.030875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-dshm\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:46.031126 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.031105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-tls-certs\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:46.036374 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.036355    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bst2d\" (UniqueName: \"kubernetes.io/projected/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-kube-api-access-bst2d\") pod \"premium-w4-ee96ec-kserve-7c88698c67-sg8ch\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") " pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:46.057951 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.057927    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:46.150068 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.150034    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:46.422708 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.422630    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"]
Sep 15 12:41:46.423649 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:41:46.423624    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bd21233_aa3a_48d6_8e3e_5ecc3fe79170.slice/crio-d73177bf26eca159e76841c821e1b062fc6529cdd1f9a898d9a4cef239ca9b79 WatchSource:0}: Error finding container d73177bf26eca159e76841c821e1b062fc6529cdd1f9a898d9a4cef239ca9b79: Status 404 returned error can't find the container with id d73177bf26eca159e76841c821e1b062fc6529cdd1f9a898d9a4cef239ca9b79
Sep 15 12:41:46.774265 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.774160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch" event={"ID":"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170","Type":"ContainerStarted","Data":"3d98ae5fd23c5e19bfb41e47aa58c099a7bd13a793b405972c3ba704f5a6a283"}
Sep 15 12:41:46.774265 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.774208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch" event={"ID":"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170","Type":"ContainerStarted","Data":"d73177bf26eca159e76841c821e1b062fc6529cdd1f9a898d9a4cef239ca9b79"}
Sep 15 12:41:46.791508 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:46.791445    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch" podStartSLOduration=1.7914308970000001 podStartE2EDuration="1.791430897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:41:46.789491483 +0000 UTC m=+978.174202380" watchObservedRunningTime="2026-09-15 12:41:46.791430897 +0000 UTC m=+978.176141805"
Sep 15 12:41:48.292924 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:48.292890    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:48.405765 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:48.405712    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:48.498796 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:48.498759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:41:49.766888 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:49.766859    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:41:51.565950 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:51.565866    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:41:56.150624 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:56.150588    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:56.151112 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:56.150648    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:56.163042 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:56.163019    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:41:56.841666 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:41:56.841638    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:42:02.786433 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:02.786395    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:03.608312 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:03.608271    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:04.339172 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:04.339136    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:04.439579 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:04.439548    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:05.227440 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:05.227404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:06.109184 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:06.109152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:06.890689 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:06.890658    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:07.467938 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:07.467902    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:10.055590 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:10.055554    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:10.159931 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:10.159893    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:10.259489 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:10.259450    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:17.075303 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:17.075268    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:17.128556 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:17.128517    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:19.763780 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.763738    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"]
Sep 15 12:42:19.767491 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.767469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.770057 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.770033    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-11f865\"/\"premium-w0-11f865-kserve-self-signed-certs\""
Sep 15 12:42:19.777923 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.777899    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"]
Sep 15 12:42:19.799610 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.799580    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:19.837434 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.837399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-tmp-dir\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.837616 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.837455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw8wc\" (UniqueName: \"kubernetes.io/projected/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-kube-api-access-sw8wc\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.837616 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.837524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-home\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.837747 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.837627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-dshm\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.837747 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.837661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-model-cache\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.837848 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.837755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-tls-certs\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.938380 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.938341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw8wc\" (UniqueName: \"kubernetes.io/projected/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-kube-api-access-sw8wc\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.938599 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.938394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-home\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.938599 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.938453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-dshm\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.938599 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.938475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-model-cache\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.938599 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.938530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-tls-certs\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.938599 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.938572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-tmp-dir\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.938922 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.938899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-home\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.939003 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.938947    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-model-cache\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.939003 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.938979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-tmp-dir\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.940858 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.940823    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-dshm\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.941153 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.941136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-tls-certs\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:19.945872 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:19.945850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw8wc\" (UniqueName: \"kubernetes.io/projected/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-kube-api-access-sw8wc\") pod \"premium-w0-11f865-kserve-dd78c4cc9-gvhr8\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") " pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:20.078681 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:20.078579    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:20.347401 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:20.347375    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"]
Sep 15 12:42:20.829474 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:20.829420    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:20.967379 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:20.967345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8" event={"ID":"31ba55e7-ef19-4ade-935a-d4d0ed246c3a","Type":"ContainerStarted","Data":"125a5bbf0363b35afe35d790eba26d58cd334f08a76343f28f62d9eeefcaac12"}
Sep 15 12:42:20.967555 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:20.967388    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8" event={"ID":"31ba55e7-ef19-4ade-935a-d4d0ed246c3a","Type":"ContainerStarted","Data":"306a5b9674b99554ad5d40692c246713501be3c295846c2b1d51d09d244df7ab"}
Sep 15 12:42:20.983974 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:20.983927    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8" podStartSLOduration=1.983913425 podStartE2EDuration="1.983913425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:42:20.982695568 +0000 UTC m=+1012.367406462" watchObservedRunningTime="2026-09-15 12:42:20.983913425 +0000 UTC m=+1012.368624308"
Sep 15 12:42:21.172701 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:21.172668    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:21.475233 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:21.475148    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:24.499402 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:24.499367    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:25.665026 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:25.664986    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:42:27.379696 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:27.379653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:27.438605 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:27.438571    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:27.514403 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:27.514354    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:27.615990 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:27.615952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:27.711876 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:27.711767    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:28.320527 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:28.320485    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:28.422544 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:28.422502    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:28.520260 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:28.520185    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:30.018376 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:30.018338    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:30.079138 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:30.079110    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:30.079308 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:30.079156    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:30.091991 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:30.091958    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:31.037631 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:31.037598    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:42:35.906303 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:35.906268    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:36.006318 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:36.006276    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:36.570303 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:36.570267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:43.461692 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:43.461651    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:43.555395 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:43.555361    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:47.778415 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:47.778378    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:48.354317 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.354283    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:48.449224 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.449182    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:48.547470 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.547421    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:48.642735 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.642657    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"]
Sep 15 12:42:48.646207 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.646191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.649041 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.649017    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0f955e\"/\"openshift-service-ca.crt\""
Sep 15 12:42:48.649041 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.649021    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0f955e\"/\"kube-root-ca.crt\""
Sep 15 12:42:48.650112 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.650097    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0f955e\"/\"default-dockercfg-gdfg5\""
Sep 15 12:42:48.650259 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.650159    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0f955e\"/\"explicit-model-0f955e-kserve-self-signed-certs\""
Sep 15 12:42:48.655929 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.655907    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"]
Sep 15 12:42:48.707671 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.707641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-dshm\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.707809 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.707692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c141d633-5084-4979-a48a-68349c0df53e-tls-certs\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.707809 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.707749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqfwt\" (UniqueName: \"kubernetes.io/projected/c141d633-5084-4979-a48a-68349c0df53e-kube-api-access-gqfwt\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.707932 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.707870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-model-cache\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.707932 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.707905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-tmp-dir\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.707932 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.707921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-home\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.809231 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.809190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-tmp-dir\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.809231 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.809232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-home\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.809749 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.809281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-dshm\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.809749 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.809298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c141d633-5084-4979-a48a-68349c0df53e-tls-certs\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.809749 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.809321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqfwt\" (UniqueName: \"kubernetes.io/projected/c141d633-5084-4979-a48a-68349c0df53e-kube-api-access-gqfwt\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.809749 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.809392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-model-cache\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.809749 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.809595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-tmp-dir\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.810006 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.809763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-model-cache\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.810006 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.809804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-home\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.811655 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.811628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-dshm\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.811901 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.811886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c141d633-5084-4979-a48a-68349c0df53e-tls-certs\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.820594 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.820573    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqfwt\" (UniqueName: \"kubernetes.io/projected/c141d633-5084-4979-a48a-68349c0df53e-kube-api-access-gqfwt\") pod \"explicit-model-0f955e-kserve-66b79f9cdf-sdhdx\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") " pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.955770 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.955690    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:48.969874 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:48.969822    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:42:49.230348 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:49.230325    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"]
Sep 15 12:42:49.232212 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:42:49.232190    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc141d633_5084_4979_a48a_68349c0df53e.slice/crio-ba9300b4beb5fbab3da69a7b8996d9acf94f94923b6b1cedd2ce8852324bddcd WatchSource:0}: Error finding container ba9300b4beb5fbab3da69a7b8996d9acf94f94923b6b1cedd2ce8852324bddcd: Status 404 returned error can't find the container with id ba9300b4beb5fbab3da69a7b8996d9acf94f94923b6b1cedd2ce8852324bddcd
Sep 15 12:42:49.351382 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:49.351345    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:49.448804 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:49.448760    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:50.107821 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:50.107781    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:50.129777 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:50.129747    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx" event={"ID":"c141d633-5084-4979-a48a-68349c0df53e","Type":"ContainerStarted","Data":"369628879bf5aa670e6f9ab94ec95ecb8684bc2ccbf04ac7885113fda7df30eb"}
Sep 15 12:42:50.129944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:50.129784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx" event={"ID":"c141d633-5084-4979-a48a-68349c0df53e","Type":"ContainerStarted","Data":"ba9300b4beb5fbab3da69a7b8996d9acf94f94923b6b1cedd2ce8852324bddcd"}
Sep 15 12:42:50.148187 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:50.148144    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx" podStartSLOduration=2.148129612 podStartE2EDuration="2.148129612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:42:50.145712743 +0000 UTC m=+1041.530423630" watchObservedRunningTime="2026-09-15 12:42:50.148129612 +0000 UTC m=+1041.532840523"
Sep 15 12:42:50.873187 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:50.873134    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:51.516165 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:51.516132    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:52.209070 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:52.209038    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:52.324197 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:52.324161    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:52.462418 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:52.462327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:52.712898 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:52.712800    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:52.816470 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:52.816438    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:42:58.956903 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:58.956859    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:58.957308 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:58.956919    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:58.969058 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:58.969035    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:42:59.186984 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:42:59.186950    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:43:12.957630 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:12.957593    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:43:14.288549 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:14.288491    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:14.966565 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:14.966528    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:15.791076 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:15.791038    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:16.476635 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:16.476599    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:17.290330 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:17.290292    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:17.989483 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:17.989451    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:18.411510 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:18.411472    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:18.778210 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:18.778128    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:19.045779 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:19.045689    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:19.143591 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:19.143560    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:20.488909 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:20.488867    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:21.065357 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:21.065325    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:24.810328 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:24.810238    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:25.504039 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:25.504010    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:25.653110 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:25.653077    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:25.732362 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:25.732328    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:26.763274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:26.763234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:27.178541 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:27.178511    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:27.988445 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:27.988407    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:28.631203 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:28.631171    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:29.272888 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:29.272850    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:29.991602 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:29.991557    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:30.678845 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:30.678801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:31.481494 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:31.481458    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:32.553333 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:32.553300    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:43:34.762348 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:34.762309    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:35.296655 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.296622    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"]
Sep 15 12:43:35.296886 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.296821    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2" podUID="c670ef38-d3c6-48e5-be9d-c9734b0b979c" containerName="maas-api" containerID="cri-o://4e16caf80dcb9e882cf011b12757ff3027ecd99a8f99babbad6e25dcad1e30ce" gracePeriod=30
Sep 15 12:43:35.328869 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.328813    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:35.429205 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.429166    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:35.508280 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.508247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:35.536257 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.536235    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:43:35.569017 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.568933    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:35.647133 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.647100    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c670ef38-d3c6-48e5-be9d-c9734b0b979c-openshift-service-ca\") pod \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") "
Sep 15 12:43:35.647315 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.647195    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-metrics-certs\") pod \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") "
Sep 15 12:43:35.647315 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.647237    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-maas-api-tls\") pod \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") "
Sep 15 12:43:35.647315 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.647286    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh5t7\" (UniqueName: \"kubernetes.io/projected/c670ef38-d3c6-48e5-be9d-c9734b0b979c-kube-api-access-lh5t7\") pod \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\" (UID: \"c670ef38-d3c6-48e5-be9d-c9734b0b979c\") "
Sep 15 12:43:35.647498 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.647464    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c670ef38-d3c6-48e5-be9d-c9734b0b979c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c670ef38-d3c6-48e5-be9d-c9734b0b979c" (UID: "c670ef38-d3c6-48e5-be9d-c9734b0b979c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:43:35.647575 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.647558    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c670ef38-d3c6-48e5-be9d-c9734b0b979c-openshift-service-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:43:35.649385 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.649346    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c670ef38-d3c6-48e5-be9d-c9734b0b979c" (UID: "c670ef38-d3c6-48e5-be9d-c9734b0b979c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:43:35.649385 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.649375    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c670ef38-d3c6-48e5-be9d-c9734b0b979c" (UID: "c670ef38-d3c6-48e5-be9d-c9734b0b979c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:43:35.649543 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.649449    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c670ef38-d3c6-48e5-be9d-c9734b0b979c-kube-api-access-lh5t7" (OuterVolumeSpecName: "kube-api-access-lh5t7") pod "c670ef38-d3c6-48e5-be9d-c9734b0b979c" (UID: "c670ef38-d3c6-48e5-be9d-c9734b0b979c"). InnerVolumeSpecName "kube-api-access-lh5t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:43:35.749070 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.749035    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-metrics-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:43:35.749070 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.749068    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c670ef38-d3c6-48e5-be9d-c9734b0b979c-maas-api-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:43:35.749297 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.749081    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh5t7\" (UniqueName: \"kubernetes.io/projected/c670ef38-d3c6-48e5-be9d-c9734b0b979c-kube-api-access-lh5t7\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:43:35.986308 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:35.986274    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:36.086445 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:36.086406    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:36.377347 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:36.377315    2579 generic.go:356] "Generic (PLEG): container finished" podID="c670ef38-d3c6-48e5-be9d-c9734b0b979c" containerID="4e16caf80dcb9e882cf011b12757ff3027ecd99a8f99babbad6e25dcad1e30ce" exitCode=0
Sep 15 12:43:36.377534 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:36.377359    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"
Sep 15 12:43:36.377534 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:36.377367    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2" event={"ID":"c670ef38-d3c6-48e5-be9d-c9734b0b979c","Type":"ContainerDied","Data":"4e16caf80dcb9e882cf011b12757ff3027ecd99a8f99babbad6e25dcad1e30ce"}
Sep 15 12:43:36.377534 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:36.377415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2" event={"ID":"c670ef38-d3c6-48e5-be9d-c9734b0b979c","Type":"ContainerDied","Data":"608ac513a65b64b0c674ec4942682aa32c41f9bbae8ca33d3fd0a1ad3db7eb21"}
Sep 15 12:43:36.377534 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:36.377439    2579 scope.go:117] "RemoveContainer" containerID="4e16caf80dcb9e882cf011b12757ff3027ecd99a8f99babbad6e25dcad1e30ce"
Sep 15 12:43:36.387618 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:36.387602    2579 scope.go:117] "RemoveContainer" containerID="4e16caf80dcb9e882cf011b12757ff3027ecd99a8f99babbad6e25dcad1e30ce"
Sep 15 12:43:36.387917 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:43:36.387891    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e16caf80dcb9e882cf011b12757ff3027ecd99a8f99babbad6e25dcad1e30ce\": container with ID starting with 4e16caf80dcb9e882cf011b12757ff3027ecd99a8f99babbad6e25dcad1e30ce not found: ID does not exist" containerID="4e16caf80dcb9e882cf011b12757ff3027ecd99a8f99babbad6e25dcad1e30ce"
Sep 15 12:43:36.387977 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:36.387930    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e16caf80dcb9e882cf011b12757ff3027ecd99a8f99babbad6e25dcad1e30ce"} err="failed to get container status \"4e16caf80dcb9e882cf011b12757ff3027ecd99a8f99babbad6e25dcad1e30ce\": rpc error: code = NotFound desc = could not find container \"4e16caf80dcb9e882cf011b12757ff3027ecd99a8f99babbad6e25dcad1e30ce\": container with ID starting with 4e16caf80dcb9e882cf011b12757ff3027ecd99a8f99babbad6e25dcad1e30ce not found: ID does not exist"
Sep 15 12:43:36.399495 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:36.399469    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"]
Sep 15 12:43:36.401661 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:36.401635    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f955e-5b5bcd797-lpsn2"]
Sep 15 12:43:36.796038 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:36.795946    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:37.105997 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:37.105971    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c670ef38-d3c6-48e5-be9d-c9734b0b979c" path="/var/lib/kubelet/pods/c670ef38-d3c6-48e5-be9d-c9734b0b979c/volumes"
Sep 15 12:43:37.448881 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:37.448766    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:38.110120 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:38.110087    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:38.903511 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:38.903473    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:39.479046 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:39.479018    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:39.579564 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:39.579534    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:39.675273 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:39.675235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:39.735298 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:39.735210    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:40.227547 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:40.227512    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:40.573906 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:40.573779    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:40.842690 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:40.842610    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"]
Sep 15 12:43:41.073887 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:41.073851    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:41.288510 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:41.288471    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:42.260681 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:42.260650    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:42.768959 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:42.768919    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:45.827050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:45.827010    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:45.927790 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:45.927756    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:46.027132 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:46.027097    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:46.101174 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:46.101128    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:46.434647 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:46.434569    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:46.531267 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:46.531196    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:46.647128 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:46.647088    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:51.757178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.757142    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"]
Sep 15 12:43:51.757574 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.757558    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c670ef38-d3c6-48e5-be9d-c9734b0b979c" containerName="maas-api"
Sep 15 12:43:51.757628 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.757575    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c670ef38-d3c6-48e5-be9d-c9734b0b979c" containerName="maas-api"
Sep 15 12:43:51.757665 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.757641    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c670ef38-d3c6-48e5-be9d-c9734b0b979c" containerName="maas-api"
Sep 15 12:43:51.764409 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.764388    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.768712 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.768678    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ee96ec\"/\"e2e-conflict-a-2381886b-kserve-self-signed-certs\""
Sep 15 12:43:51.771184 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.771160    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"]
Sep 15 12:43:51.791609 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.791582    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:51.896699 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.896666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/478306d3-9380-4336-b300-9d9f313eb0e1-tls-certs\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.896859 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.896726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-tmp-dir\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.896859 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.896781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v9bk\" (UniqueName: \"kubernetes.io/projected/478306d3-9380-4336-b300-9d9f313eb0e1-kube-api-access-5v9bk\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.896859 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.896810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-dshm\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.896970 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.896924    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-model-cache\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.897008 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.896964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-home\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.997562 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.997530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-home\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.997712 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.997577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/478306d3-9380-4336-b300-9d9f313eb0e1-tls-certs\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.997712 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.997699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-tmp-dir\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.997782 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.997766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v9bk\" (UniqueName: \"kubernetes.io/projected/478306d3-9380-4336-b300-9d9f313eb0e1-kube-api-access-5v9bk\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.997855 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.997814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-dshm\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.997955 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.997928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-home\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.997955 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.997935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-model-cache\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.998075 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.998058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-tmp-dir\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.998213 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.998196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-model-cache\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:51.999932 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:51.999910    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-dshm\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:52.000104 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:52.000088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/478306d3-9380-4336-b300-9d9f313eb0e1-tls-certs\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:52.005532 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:52.005512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v9bk\" (UniqueName: \"kubernetes.io/projected/478306d3-9380-4336-b300-9d9f313eb0e1-kube-api-access-5v9bk\") pod \"e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:52.078010 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:52.077935    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:43:52.351145 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:52.351118    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"]
Sep 15 12:43:52.353081 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:43:52.353053    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod478306d3_9380_4336_b300_9d9f313eb0e1.slice/crio-f62465d6b9185ee492256d6548f17ce38795f5d72f641ddecc674fb721a78cf9 WatchSource:0}: Error finding container f62465d6b9185ee492256d6548f17ce38795f5d72f641ddecc674fb721a78cf9: Status 404 returned error can't find the container with id f62465d6b9185ee492256d6548f17ce38795f5d72f641ddecc674fb721a78cf9
Sep 15 12:43:52.462146 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:52.462110    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7" event={"ID":"478306d3-9380-4336-b300-9d9f313eb0e1","Type":"ContainerStarted","Data":"3a2a90f29fae653391b134f0f2f7d649157a5ab15414ad1c675af7633bb75100"}
Sep 15 12:43:52.462340 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:52.462166    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7" event={"ID":"478306d3-9380-4336-b300-9d9f313eb0e1","Type":"ContainerStarted","Data":"f62465d6b9185ee492256d6548f17ce38795f5d72f641ddecc674fb721a78cf9"}
Sep 15 12:43:52.478570 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:52.478517    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7" podStartSLOduration=1.478497259 podStartE2EDuration="1.478497259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:43:52.477754423 +0000 UTC m=+1103.862465309" watchObservedRunningTime="2026-09-15 12:43:52.478497259 +0000 UTC m=+1103.863208144"
Sep 15 12:43:52.609488 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:52.609397    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:52.713614 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:52.713568    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:52.809761 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:52.809721    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:52.935695 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:52.935616    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:54.609209 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:54.609177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:55.861821 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:55.861777    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx" podUID="c141d633-5084-4979-a48a-68349c0df53e" containerName="main" containerID="cri-o://369628879bf5aa670e6f9ab94ec95ecb8684bc2ccbf04ac7885113fda7df30eb" gracePeriod=45
Sep 15 12:43:56.113401 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.113343    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:43:56.136883 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.136852    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c141d633-5084-4979-a48a-68349c0df53e-tls-certs\") pod \"c141d633-5084-4979-a48a-68349c0df53e\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") "
Sep 15 12:43:56.137013 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.136945    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqfwt\" (UniqueName: \"kubernetes.io/projected/c141d633-5084-4979-a48a-68349c0df53e-kube-api-access-gqfwt\") pod \"c141d633-5084-4979-a48a-68349c0df53e\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") "
Sep 15 12:43:56.137013 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.137003    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-home\") pod \"c141d633-5084-4979-a48a-68349c0df53e\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") "
Sep 15 12:43:56.137118 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.137091    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-tmp-dir\") pod \"c141d633-5084-4979-a48a-68349c0df53e\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") "
Sep 15 12:43:56.137201 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.137150    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-dshm\") pod \"c141d633-5084-4979-a48a-68349c0df53e\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") "
Sep 15 12:43:56.137201 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.137187    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-model-cache\") pod \"c141d633-5084-4979-a48a-68349c0df53e\" (UID: \"c141d633-5084-4979-a48a-68349c0df53e\") "
Sep 15 12:43:56.137314 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.137257    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-home" (OuterVolumeSpecName: "home") pod "c141d633-5084-4979-a48a-68349c0df53e" (UID: "c141d633-5084-4979-a48a-68349c0df53e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:43:56.137395 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.137358    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c141d633-5084-4979-a48a-68349c0df53e" (UID: "c141d633-5084-4979-a48a-68349c0df53e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:43:56.137586 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.137563    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-model-cache" (OuterVolumeSpecName: "model-cache") pod "c141d633-5084-4979-a48a-68349c0df53e" (UID: "c141d633-5084-4979-a48a-68349c0df53e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:43:56.137681 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.137664    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:43:56.137748 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.137683    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:43:56.137748 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.137696    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:43:56.139128 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.139098    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-dshm" (OuterVolumeSpecName: "dshm") pod "c141d633-5084-4979-a48a-68349c0df53e" (UID: "c141d633-5084-4979-a48a-68349c0df53e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:43:56.139249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.139207    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c141d633-5084-4979-a48a-68349c0df53e-kube-api-access-gqfwt" (OuterVolumeSpecName: "kube-api-access-gqfwt") pod "c141d633-5084-4979-a48a-68349c0df53e" (UID: "c141d633-5084-4979-a48a-68349c0df53e"). InnerVolumeSpecName "kube-api-access-gqfwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:43:56.139249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.139240    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c141d633-5084-4979-a48a-68349c0df53e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c141d633-5084-4979-a48a-68349c0df53e" (UID: "c141d633-5084-4979-a48a-68349c0df53e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:43:56.238306 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.238270    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c141d633-5084-4979-a48a-68349c0df53e-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:43:56.238306 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.238299    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqfwt\" (UniqueName: \"kubernetes.io/projected/c141d633-5084-4979-a48a-68349c0df53e-kube-api-access-gqfwt\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:43:56.238306 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.238308    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c141d633-5084-4979-a48a-68349c0df53e-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:43:56.484494 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.484406    2579 generic.go:356] "Generic (PLEG): container finished" podID="c141d633-5084-4979-a48a-68349c0df53e" containerID="369628879bf5aa670e6f9ab94ec95ecb8684bc2ccbf04ac7885113fda7df30eb" exitCode=0
Sep 15 12:43:56.484494 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.484454    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"
Sep 15 12:43:56.484711 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.484490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx" event={"ID":"c141d633-5084-4979-a48a-68349c0df53e","Type":"ContainerDied","Data":"369628879bf5aa670e6f9ab94ec95ecb8684bc2ccbf04ac7885113fda7df30eb"}
Sep 15 12:43:56.484711 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.484534    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx" event={"ID":"c141d633-5084-4979-a48a-68349c0df53e","Type":"ContainerDied","Data":"ba9300b4beb5fbab3da69a7b8996d9acf94f94923b6b1cedd2ce8852324bddcd"}
Sep 15 12:43:56.484711 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.484551    2579 scope.go:117] "RemoveContainer" containerID="369628879bf5aa670e6f9ab94ec95ecb8684bc2ccbf04ac7885113fda7df30eb"
Sep 15 12:43:56.493925 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.493910    2579 scope.go:117] "RemoveContainer" containerID="369628879bf5aa670e6f9ab94ec95ecb8684bc2ccbf04ac7885113fda7df30eb"
Sep 15 12:43:56.494178 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:43:56.494159    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"369628879bf5aa670e6f9ab94ec95ecb8684bc2ccbf04ac7885113fda7df30eb\": container with ID starting with 369628879bf5aa670e6f9ab94ec95ecb8684bc2ccbf04ac7885113fda7df30eb not found: ID does not exist" containerID="369628879bf5aa670e6f9ab94ec95ecb8684bc2ccbf04ac7885113fda7df30eb"
Sep 15 12:43:56.494245 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.494189    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"369628879bf5aa670e6f9ab94ec95ecb8684bc2ccbf04ac7885113fda7df30eb"} err="failed to get container status \"369628879bf5aa670e6f9ab94ec95ecb8684bc2ccbf04ac7885113fda7df30eb\": rpc error: code = NotFound desc = could not find container \"369628879bf5aa670e6f9ab94ec95ecb8684bc2ccbf04ac7885113fda7df30eb\": container with ID starting with 369628879bf5aa670e6f9ab94ec95ecb8684bc2ccbf04ac7885113fda7df30eb not found: ID does not exist"
Sep 15 12:43:56.505600 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.505579    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"]
Sep 15 12:43:56.511355 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:56.511335    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-0f955e/explicit-model-0f955e-kserve-66b79f9cdf-sdhdx"]
Sep 15 12:43:57.105240 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:57.105206    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c141d633-5084-4979-a48a-68349c0df53e" path="/var/lib/kubelet/pods/c141d633-5084-4979-a48a-68349c0df53e/volumes"
Sep 15 12:43:57.958926 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:57.958890    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:43:58.569776 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:58.569742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:58.964898 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:58.964844    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:59.173337 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:59.173293    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:43:59.279783 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:43:59.279677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:00.134472 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:00.134431    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:00.431115 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:00.431032    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:00.614500 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:00.614468    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:02.078390 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:02.078355    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:44:02.078785 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:02.078412    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:44:02.090786 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:02.090763    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:44:02.527671 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:02.527639    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:44:03.201412 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:03.201368    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:03.890226 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:03.890194    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:03.989237 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:03.989186    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:04.181631 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:04.181547    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:06.405903 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:06.405860    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:06.511960 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:06.511921    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:06.606569 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:06.606533    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:07.276764 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.276728    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"]
Sep 15 12:44:07.277327 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.277298    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c141d633-5084-4979-a48a-68349c0df53e" containerName="main"
Sep 15 12:44:07.277327 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.277323    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c141d633-5084-4979-a48a-68349c0df53e" containerName="main"
Sep 15 12:44:07.277484 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.277470    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c141d633-5084-4979-a48a-68349c0df53e" containerName="main"
Sep 15 12:44:07.281251 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.281230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:07.283670 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.283644    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-a1ba7b-serving-cert\""
Sep 15 12:44:07.289458 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.289435    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"]
Sep 15 12:44:07.308982 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.308952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:07.336812 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.336778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:07.336999 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.336823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:07.336999 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.336941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:07.336999 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.336988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7hzr\" (UniqueName: \"kubernetes.io/projected/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-kube-api-access-d7hzr\") pod \"maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:07.369403 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.369364    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:07.437588 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.437553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7hzr\" (UniqueName: \"kubernetes.io/projected/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-kube-api-access-d7hzr\") pod \"maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:07.438114 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.437600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:07.438114 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.437639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:07.438114 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.437758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:07.438114 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:44:07.437946    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-serving-cert: secret "maas-api-e2e-shared-a-w2-a1ba7b-serving-cert" not found
Sep 15 12:44:07.438114 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:44:07.438030    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-maas-api-tls podName:afd23e0f-7668-41d6-8f66-a8bffa2cc1f2 nodeName:}" failed. No retries permitted until 2026-09-15 12:44:07.938009513 +0000 UTC m=+1119.322720375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-maas-api-tls") pod "maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p" (UID: "afd23e0f-7668-41d6-8f66-a8bffa2cc1f2") : secret "maas-api-e2e-shared-a-w2-a1ba7b-serving-cert" not found
Sep 15 12:44:07.438396 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.438377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:07.440113 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.440094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:07.448569 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.448546    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7hzr\" (UniqueName: \"kubernetes.io/projected/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-kube-api-access-d7hzr\") pod \"maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:07.763530 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.763493    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:07.866861 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.866814    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:07.943773 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:07.943734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:07.943961 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:44:07.943894    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-serving-cert: secret "maas-api-e2e-shared-a-w2-a1ba7b-serving-cert" not found
Sep 15 12:44:07.943961 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:44:07.943959    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-maas-api-tls podName:afd23e0f-7668-41d6-8f66-a8bffa2cc1f2 nodeName:}" failed. No retries permitted until 2026-09-15 12:44:08.943942945 +0000 UTC m=+1120.328653812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-maas-api-tls") pod "maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p" (UID: "afd23e0f-7668-41d6-8f66-a8bffa2cc1f2") : secret "maas-api-e2e-shared-a-w2-a1ba7b-serving-cert" not found
Sep 15 12:44:08.080960 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:08.080864    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:08.957892 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:08.957854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:08.960307 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:08.960287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:09.095002 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:09.094964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:09.368791 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:09.368766    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"]
Sep 15 12:44:09.370455 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:44:09.370425    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd23e0f_7668_41d6_8f66_a8bffa2cc1f2.slice/crio-0779084d6e5a3c714e9d0291e4afec55e8a5367d719836c89f99a9edcdba2308 WatchSource:0}: Error finding container 0779084d6e5a3c714e9d0291e4afec55e8a5367d719836c89f99a9edcdba2308: Status 404 returned error can't find the container with id 0779084d6e5a3c714e9d0291e4afec55e8a5367d719836c89f99a9edcdba2308
Sep 15 12:44:09.554879 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:09.554823    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p" event={"ID":"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2","Type":"ContainerStarted","Data":"0779084d6e5a3c714e9d0291e4afec55e8a5367d719836c89f99a9edcdba2308"}
Sep 15 12:44:10.562091 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:10.562048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p" event={"ID":"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2","Type":"ContainerStarted","Data":"2de0bf4ae8a36e4d3eafd4ddaafaa253b5b16551f64dddde45b2acde6923c6d9"}
Sep 15 12:44:10.562480 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:10.562120    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:10.579735 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:10.579686    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p" podStartSLOduration=3.104975408 podStartE2EDuration="3.579669923s" totalImagesPullingTime="474.694515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:44:10.577482939 +0000 UTC m=+1121.962193825" watchObservedRunningTime="2026-09-15 12:44:10.579669923 +0000 UTC m=+1121.964380851"
Sep 15 12:44:16.573528 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:16.573498    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:44:17.872367 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:17.872338    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:18.129788 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:18.129697    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:18.616650 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:18.616621    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:18.932276 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:18.932175    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:19.032704 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:19.032661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:20.365515 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:20.365477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:20.464499 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:20.464463    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:20.586223 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:20.586178    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:23.489276 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:23.489236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:28.292798 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:28.292762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:29.581137 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:29.581039    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:30.367945 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:30.367902    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:31.186818 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:31.186783    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:32.016340 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:32.016300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:32.719715 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:32.719683    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:33.390853 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:33.390796    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:34.128382 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:34.128346    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:34.158890 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:34.158856    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:44:34.332434 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:34.332394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:34.658340 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:34.658305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:35.853316 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:35.853283    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:36.021326 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:36.021289    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:36.987969 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:36.987929    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:37.579807 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:37.579776    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:37.681112 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:37.681076    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:37.780661 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:37.780623    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:38.544052 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:38.544018    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:39.369388 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:39.369350    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:40.106794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:40.106741    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:40.874510 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:40.874477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:41.724909 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:41.724864    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:41.824566 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:41.824531    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:42.200352 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:42.200303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:44:42.301637 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:44:42.301602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:00.176132 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.176047    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e3fef-29824605-bjzr6"]
Sep 15 12:45:00.179385 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.179363    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824605-9j559"]
Sep 15 12:45:00.179506 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.179481    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e3fef-29824605-bjzr6"
Sep 15 12:45:00.182065 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.182042    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\""
Sep 15 12:45:00.182570 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.182556    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824605-9j559"
Sep 15 12:45:00.193595 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.193573    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e3fef-29824605-bjzr6"
Sep 15 12:45:00.194730 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.194705    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e3fef-29824605-bjzr6"]
Sep 15 12:45:00.200669 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.200648    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824605-9j559"
Sep 15 12:45:00.202236 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.202212    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ee96ec-29824605-lt7p8"]
Sep 15 12:45:00.205594 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.205575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ee96ec-29824605-lt7p8"
Sep 15 12:45:00.224671 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.224648    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824605-9j559"]
Sep 15 12:45:00.231488 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.231425    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ee96ec-29824605-lt7p8"]
Sep 15 12:45:00.259503 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.259452    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ee96ec-29824605-lt7p8"
Sep 15 12:45:00.477776 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.477751    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e3fef-29824605-bjzr6"]
Sep 15 12:45:00.479372 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:45:00.479342    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c787bc6_85d8_42f9_9e75_332a69028fa4.slice/crio-ee0a6bb7f8c0b7b88239991c2698b01c623188f69c46e7469e40e5c3c725d410 WatchSource:0}: Error finding container ee0a6bb7f8c0b7b88239991c2698b01c623188f69c46e7469e40e5c3c725d410: Status 404 returned error can't find the container with id ee0a6bb7f8c0b7b88239991c2698b01c623188f69c46e7469e40e5c3c725d410
Sep 15 12:45:00.512798 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.512769    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824605-9j559"]
Sep 15 12:45:00.514397 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:45:00.514372    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded29b541_4aac_4143_b8fd_6c0b6c160feb.slice/crio-fd5ec20bf5c6965fa5087cc6d18e990967e28d8a55ddfb1554577886aec94551 WatchSource:0}: Error finding container fd5ec20bf5c6965fa5087cc6d18e990967e28d8a55ddfb1554577886aec94551: Status 404 returned error can't find the container with id fd5ec20bf5c6965fa5087cc6d18e990967e28d8a55ddfb1554577886aec94551
Sep 15 12:45:00.543840 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.543804    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ee96ec-29824605-lt7p8"]
Sep 15 12:45:00.546092 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:45:00.546067    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcedf67b2_3bce_4f30_b0d5_9a7a195ca694.slice/crio-6e77495f06a99993377c96e0d583a73c6f12f7e4b6019f9163f23514bb5d7b49 WatchSource:0}: Error finding container 6e77495f06a99993377c96e0d583a73c6f12f7e4b6019f9163f23514bb5d7b49: Status 404 returned error can't find the container with id 6e77495f06a99993377c96e0d583a73c6f12f7e4b6019f9163f23514bb5d7b49
Sep 15 12:45:00.834370 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.834288    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824605-9j559" event={"ID":"ed29b541-4aac-4143-b8fd-6c0b6c160feb","Type":"ContainerStarted","Data":"fd5ec20bf5c6965fa5087cc6d18e990967e28d8a55ddfb1554577886aec94551"}
Sep 15 12:45:00.835575 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.835543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ee96ec-29824605-lt7p8" event={"ID":"cedf67b2-3bce-4f30-b0d5-9a7a195ca694","Type":"ContainerStarted","Data":"6e77495f06a99993377c96e0d583a73c6f12f7e4b6019f9163f23514bb5d7b49"}
Sep 15 12:45:00.836824 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:00.836802    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e3fef-29824605-bjzr6" event={"ID":"9c787bc6-85d8-42f9-9e75-332a69028fa4","Type":"ContainerStarted","Data":"ee0a6bb7f8c0b7b88239991c2698b01c623188f69c46e7469e40e5c3c725d410"}
Sep 15 12:45:03.857928 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:03.857889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824605-9j559" event={"ID":"ed29b541-4aac-4143-b8fd-6c0b6c160feb","Type":"ContainerStarted","Data":"e6082b6588feb8f3483960a7c06d6bb259786d3b10cf0107d786ef1a5531855e"}
Sep 15 12:45:03.859815 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:03.859788    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ee96ec-29824605-lt7p8" event={"ID":"cedf67b2-3bce-4f30-b0d5-9a7a195ca694","Type":"ContainerStarted","Data":"bd716cac22af1f2392c23d56a990c4e61e60dcbb7308c4d7690450607013d5af"}
Sep 15 12:45:03.861387 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:03.861361    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e3fef-29824605-bjzr6" event={"ID":"9c787bc6-85d8-42f9-9e75-332a69028fa4","Type":"ContainerStarted","Data":"da1b86020c88720e525aac7f0557a8c932dd58f7be548ef08fc92ae3b66b503b"}
Sep 15 12:45:03.874881 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:03.874818    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824605-9j559" podStartSLOduration=1.384170297 podStartE2EDuration="3.874801349s" totalImagesPullingTime="2.490631052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:45:03.873921267 +0000 UTC m=+1175.258632150" watchObservedRunningTime="2026-09-15 12:45:03.874801349 +0000 UTC m=+1175.259512234"
Sep 15 12:45:03.888276 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:03.888229    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e3fef-29824605-bjzr6" podStartSLOduration=1.361739469 podStartE2EDuration="3.88821354s" totalImagesPullingTime="2.526474071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:45:03.887059359 +0000 UTC m=+1175.271770244" watchObservedRunningTime="2026-09-15 12:45:03.88821354 +0000 UTC m=+1175.272924423"
Sep 15 12:45:03.902752 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:03.902708    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ee96ec-29824605-lt7p8" podStartSLOduration=1.443542672 podStartE2EDuration="3.902692952s" totalImagesPullingTime="2.45915028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:45:03.900314835 +0000 UTC m=+1175.285025724" watchObservedRunningTime="2026-09-15 12:45:03.902692952 +0000 UTC m=+1175.287403839"
Sep 15 12:45:03.907892 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:03.907862    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:04.823504 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:04.823474    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:04.869330 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:04.869296    2579 generic.go:356] "Generic (PLEG): container finished" podID="cedf67b2-3bce-4f30-b0d5-9a7a195ca694" containerID="bd716cac22af1f2392c23d56a990c4e61e60dcbb7308c4d7690450607013d5af" exitCode=0
Sep 15 12:45:04.869712 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:04.869373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ee96ec-29824605-lt7p8" event={"ID":"cedf67b2-3bce-4f30-b0d5-9a7a195ca694","Type":"ContainerDied","Data":"bd716cac22af1f2392c23d56a990c4e61e60dcbb7308c4d7690450607013d5af"}
Sep 15 12:45:04.870938 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:04.870917    2579 generic.go:356] "Generic (PLEG): container finished" podID="9c787bc6-85d8-42f9-9e75-332a69028fa4" containerID="da1b86020c88720e525aac7f0557a8c932dd58f7be548ef08fc92ae3b66b503b" exitCode=0
Sep 15 12:45:04.871031 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:04.870945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e3fef-29824605-bjzr6" event={"ID":"9c787bc6-85d8-42f9-9e75-332a69028fa4","Type":"ContainerDied","Data":"da1b86020c88720e525aac7f0557a8c932dd58f7be548ef08fc92ae3b66b503b"}
Sep 15 12:45:04.872578 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:04.872558    2579 generic.go:356] "Generic (PLEG): container finished" podID="ed29b541-4aac-4143-b8fd-6c0b6c160feb" containerID="e6082b6588feb8f3483960a7c06d6bb259786d3b10cf0107d786ef1a5531855e" exitCode=0
Sep 15 12:45:04.872671 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:04.872595    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824605-9j559" event={"ID":"ed29b541-4aac-4143-b8fd-6c0b6c160feb","Type":"ContainerDied","Data":"e6082b6588feb8f3483960a7c06d6bb259786d3b10cf0107d786ef1a5531855e"}
Sep 15 12:45:05.576239 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:05.576198    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:06.038292 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:06.038267    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e3fef-29824605-bjzr6"
Sep 15 12:45:06.041459 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:06.041434    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824605-9j559"
Sep 15 12:45:06.073404 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:06.073381    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ee96ec-29824605-lt7p8"
Sep 15 12:45:06.884840 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:06.884804    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824605-9j559"
Sep 15 12:45:06.884840 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:06.884820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824605-9j559" event={"ID":"ed29b541-4aac-4143-b8fd-6c0b6c160feb","Type":"ContainerDied","Data":"fd5ec20bf5c6965fa5087cc6d18e990967e28d8a55ddfb1554577886aec94551"}
Sep 15 12:45:06.885085 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:06.884890    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd5ec20bf5c6965fa5087cc6d18e990967e28d8a55ddfb1554577886aec94551"
Sep 15 12:45:06.886241 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:06.886220    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ee96ec-29824605-lt7p8"
Sep 15 12:45:06.886381 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:06.886242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ee96ec-29824605-lt7p8" event={"ID":"cedf67b2-3bce-4f30-b0d5-9a7a195ca694","Type":"ContainerDied","Data":"6e77495f06a99993377c96e0d583a73c6f12f7e4b6019f9163f23514bb5d7b49"}
Sep 15 12:45:06.886381 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:06.886287    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e77495f06a99993377c96e0d583a73c6f12f7e4b6019f9163f23514bb5d7b49"
Sep 15 12:45:06.887595 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:06.887575    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e3fef-29824605-bjzr6"
Sep 15 12:45:06.887595 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:06.887588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e3fef-29824605-bjzr6" event={"ID":"9c787bc6-85d8-42f9-9e75-332a69028fa4","Type":"ContainerDied","Data":"ee0a6bb7f8c0b7b88239991c2698b01c623188f69c46e7469e40e5c3c725d410"}
Sep 15 12:45:06.887798 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:06.887625    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee0a6bb7f8c0b7b88239991c2698b01c623188f69c46e7469e40e5c3c725d410"
Sep 15 12:45:08.430668 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.430635    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7707-aa13a3"]
Sep 15 12:45:08.431078 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.431000    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cedf67b2-3bce-4f30-b0d5-9a7a195ca694" containerName="cleanup"
Sep 15 12:45:08.431078 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.431011    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cedf67b2-3bce-4f30-b0d5-9a7a195ca694" containerName="cleanup"
Sep 15 12:45:08.431078 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.431024    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed29b541-4aac-4143-b8fd-6c0b6c160feb" containerName="cleanup"
Sep 15 12:45:08.431078 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.431030    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed29b541-4aac-4143-b8fd-6c0b6c160feb" containerName="cleanup"
Sep 15 12:45:08.431078 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.431048    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c787bc6-85d8-42f9-9e75-332a69028fa4" containerName="cleanup"
Sep 15 12:45:08.431078 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.431054    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c787bc6-85d8-42f9-9e75-332a69028fa4" containerName="cleanup"
Sep 15 12:45:08.431269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.431109    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c787bc6-85d8-42f9-9e75-332a69028fa4" containerName="cleanup"
Sep 15 12:45:08.431269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.431115    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cedf67b2-3bce-4f30-b0d5-9a7a195ca694" containerName="cleanup"
Sep 15 12:45:08.431269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.431126    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed29b541-4aac-4143-b8fd-6c0b6c160feb" containerName="cleanup"
Sep 15 12:45:08.434208 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.434181    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-aa13a3"
Sep 15 12:45:08.436844 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.436807    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dhqjr\""
Sep 15 12:45:08.440635 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.440609    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-aa13a3"]
Sep 15 12:45:08.483532 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.483491    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:08.529697 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.529664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs5md\" (UniqueName: \"kubernetes.io/projected/8616e403-25c9-4b0e-bed8-3c15fe2a9908-kube-api-access-hs5md\") pod \"test-curl-7707-aa13a3\" (UID: \"8616e403-25c9-4b0e-bed8-3c15fe2a9908\") " pod="opendatahub/test-curl-7707-aa13a3"
Sep 15 12:45:08.630483 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.630446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs5md\" (UniqueName: \"kubernetes.io/projected/8616e403-25c9-4b0e-bed8-3c15fe2a9908-kube-api-access-hs5md\") pod \"test-curl-7707-aa13a3\" (UID: \"8616e403-25c9-4b0e-bed8-3c15fe2a9908\") " pod="opendatahub/test-curl-7707-aa13a3"
Sep 15 12:45:08.639181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.639150    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs5md\" (UniqueName: \"kubernetes.io/projected/8616e403-25c9-4b0e-bed8-3c15fe2a9908-kube-api-access-hs5md\") pod \"test-curl-7707-aa13a3\" (UID: \"8616e403-25c9-4b0e-bed8-3c15fe2a9908\") " pod="opendatahub/test-curl-7707-aa13a3"
Sep 15 12:45:08.700418 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.700344    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:08.745303 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:08.745276    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-aa13a3"
Sep 15 12:45:09.016192 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:09.016164    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-aa13a3"]
Sep 15 12:45:09.018345 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:45:09.018313    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8616e403_25c9_4b0e_bed8_3c15fe2a9908.slice/crio-57e9514ba7ae2da749eb8eca590c5740aa40c9e6351a016e0829d3b0daefdf33 WatchSource:0}: Error finding container 57e9514ba7ae2da749eb8eca590c5740aa40c9e6351a016e0829d3b0daefdf33: Status 404 returned error can't find the container with id 57e9514ba7ae2da749eb8eca590c5740aa40c9e6351a016e0829d3b0daefdf33
Sep 15 12:45:09.907668 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:09.907629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-aa13a3" event={"ID":"8616e403-25c9-4b0e-bed8-3c15fe2a9908","Type":"ContainerStarted","Data":"44cce4f7e77bbfbe72e72d0aa4816503bad87879908441e9a07aa767c7bbf4ed"}
Sep 15 12:45:09.907668 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:09.907669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-aa13a3" event={"ID":"8616e403-25c9-4b0e-bed8-3c15fe2a9908","Type":"ContainerStarted","Data":"57e9514ba7ae2da749eb8eca590c5740aa40c9e6351a016e0829d3b0daefdf33"}
Sep 15 12:45:09.923480 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:09.923425    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7707-aa13a3" podStartSLOduration=1.923409529 podStartE2EDuration="1.923409529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:45:09.920971712 +0000 UTC m=+1181.305682596" watchObservedRunningTime="2026-09-15 12:45:09.923409529 +0000 UTC m=+1181.308120768"
Sep 15 12:45:10.503053 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:10.503018    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7707-aa13a3"]
Sep 15 12:45:10.507428 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:10.507387    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7707-aa13a3"]
Sep 15 12:45:10.619676 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:10.619637    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7707-453692"]
Sep 15 12:45:10.620062 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:10.620048    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8616e403-25c9-4b0e-bed8-3c15fe2a9908" containerName="test-curl-7707-aa13a3"
Sep 15 12:45:10.620110 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:10.620063    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8616e403-25c9-4b0e-bed8-3c15fe2a9908" containerName="test-curl-7707-aa13a3"
Sep 15 12:45:10.620149 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:10.620122    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8616e403-25c9-4b0e-bed8-3c15fe2a9908" containerName="test-curl-7707-aa13a3"
Sep 15 12:45:10.623449 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:10.623423    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-453692"
Sep 15 12:45:10.628608 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:10.628475    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-453692"]
Sep 15 12:45:10.749947 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:10.749915    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n56z\" (UniqueName: \"kubernetes.io/projected/81a05de4-8288-479e-b26a-9157f27df1b9-kube-api-access-8n56z\") pod \"test-curl-7707-453692\" (UID: \"81a05de4-8288-479e-b26a-9157f27df1b9\") " pod="opendatahub/test-curl-7707-453692"
Sep 15 12:45:10.775390 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:10.775359    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:10.850569 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:10.850533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n56z\" (UniqueName: \"kubernetes.io/projected/81a05de4-8288-479e-b26a-9157f27df1b9-kube-api-access-8n56z\") pod \"test-curl-7707-453692\" (UID: \"81a05de4-8288-479e-b26a-9157f27df1b9\") " pod="opendatahub/test-curl-7707-453692"
Sep 15 12:45:10.858048 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:10.858025    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n56z\" (UniqueName: \"kubernetes.io/projected/81a05de4-8288-479e-b26a-9157f27df1b9-kube-api-access-8n56z\") pod \"test-curl-7707-453692\" (UID: \"81a05de4-8288-479e-b26a-9157f27df1b9\") " pod="opendatahub/test-curl-7707-453692"
Sep 15 12:45:10.913935 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:45:10.913859    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7707-aa13a3" container="test-curl-7707-aa13a3"
Sep 15 12:45:10.942147 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:10.942122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-453692"
Sep 15 12:45:11.210807 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:11.210774    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-453692"]
Sep 15 12:45:11.215706 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:45:11.215676    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81a05de4_8288_479e_b26a_9157f27df1b9.slice/crio-92a78dc4e5a938419f39655d167dbdc0c650e5352a85aca5fede14b7eb97f4dc WatchSource:0}: Error finding container 92a78dc4e5a938419f39655d167dbdc0c650e5352a85aca5fede14b7eb97f4dc: Status 404 returned error can't find the container with id 92a78dc4e5a938419f39655d167dbdc0c650e5352a85aca5fede14b7eb97f4dc
Sep 15 12:45:11.558101 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:11.558008    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:45:11.920322 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:11.920284    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-453692" event={"ID":"81a05de4-8288-479e-b26a-9157f27df1b9","Type":"ContainerStarted","Data":"5c495be0b5b6cdb38de82e44f15e65eac14de4b4f870c67f69d20b2e144fe69b"}
Sep 15 12:45:11.920808 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:11.920334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-453692" event={"ID":"81a05de4-8288-479e-b26a-9157f27df1b9","Type":"ContainerStarted","Data":"92a78dc4e5a938419f39655d167dbdc0c650e5352a85aca5fede14b7eb97f4dc"}
Sep 15 12:45:11.920808 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:11.920313    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7707-aa13a3" podUID="8616e403-25c9-4b0e-bed8-3c15fe2a9908" containerName="test-curl-7707-aa13a3" containerID="cri-o://44cce4f7e77bbfbe72e72d0aa4816503bad87879908441e9a07aa767c7bbf4ed" gracePeriod=2
Sep 15 12:45:11.935247 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:11.935206    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7707-453692" podStartSLOduration=1.935192418 podStartE2EDuration="1.935192418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:45:11.932877084 +0000 UTC m=+1183.317587969" watchObservedRunningTime="2026-09-15 12:45:11.935192418 +0000 UTC m=+1183.319903302"
Sep 15 12:45:12.503350 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:12.503308    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7707-453692"]
Sep 15 12:45:12.506401 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:12.506364    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7707-453692"]
Sep 15 12:45:12.616066 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:12.616034    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7707-529c28"]
Sep 15 12:45:12.616409 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:12.616395    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81a05de4-8288-479e-b26a-9157f27df1b9" containerName="test-curl-7707-453692"
Sep 15 12:45:12.616454 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:12.616411    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="81a05de4-8288-479e-b26a-9157f27df1b9" containerName="test-curl-7707-453692"
Sep 15 12:45:12.616495 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:12.616486    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="81a05de4-8288-479e-b26a-9157f27df1b9" containerName="test-curl-7707-453692"
Sep 15 12:45:12.619666 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:12.619650    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-529c28"
Sep 15 12:45:12.626520 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:12.626489    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-529c28"]
Sep 15 12:45:12.668767 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:12.668739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brmhb\" (UniqueName: \"kubernetes.io/projected/c2d594c6-3fe1-4b2d-854e-4f411d7eafb1-kube-api-access-brmhb\") pod \"test-curl-7707-529c28\" (UID: \"c2d594c6-3fe1-4b2d-854e-4f411d7eafb1\") " pod="opendatahub/test-curl-7707-529c28"
Sep 15 12:45:12.769224 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:12.769188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmhb\" (UniqueName: \"kubernetes.io/projected/c2d594c6-3fe1-4b2d-854e-4f411d7eafb1-kube-api-access-brmhb\") pod \"test-curl-7707-529c28\" (UID: \"c2d594c6-3fe1-4b2d-854e-4f411d7eafb1\") " pod="opendatahub/test-curl-7707-529c28"
Sep 15 12:45:12.776813 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:12.776780    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmhb\" (UniqueName: \"kubernetes.io/projected/c2d594c6-3fe1-4b2d-854e-4f411d7eafb1-kube-api-access-brmhb\") pod \"test-curl-7707-529c28\" (UID: \"c2d594c6-3fe1-4b2d-854e-4f411d7eafb1\") " pod="opendatahub/test-curl-7707-529c28"
Sep 15 12:45:12.925641 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:45:12.925553    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7707-453692" container="test-curl-7707-453692"
Sep 15 12:45:12.930334 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:12.930317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-529c28"
Sep 15 12:45:13.206589 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:13.206562    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-529c28"]
Sep 15 12:45:13.207909 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:45:13.207882    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2d594c6_3fe1_4b2d_854e_4f411d7eafb1.slice/crio-33572ff3eba3121aedc364b6d1f42ec83125eaf1ea71da106006ec672412452e WatchSource:0}: Error finding container 33572ff3eba3121aedc364b6d1f42ec83125eaf1ea71da106006ec672412452e: Status 404 returned error can't find the container with id 33572ff3eba3121aedc364b6d1f42ec83125eaf1ea71da106006ec672412452e
Sep 15 12:45:13.933108 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:13.933082    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-529c28" event={"ID":"c2d594c6-3fe1-4b2d-854e-4f411d7eafb1","Type":"ContainerStarted","Data":"f44c0cf4a0df89e13828c58ae02edc35176e98d685bc85a18d601a08f6ebf17e"}
Sep 15 12:45:13.933481 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:13.933122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-529c28" event={"ID":"c2d594c6-3fe1-4b2d-854e-4f411d7eafb1","Type":"ContainerStarted","Data":"33572ff3eba3121aedc364b6d1f42ec83125eaf1ea71da106006ec672412452e"}
Sep 15 12:45:13.933481 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:13.933164    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7707-453692" podUID="81a05de4-8288-479e-b26a-9157f27df1b9" containerName="test-curl-7707-453692" containerID="cri-o://5c495be0b5b6cdb38de82e44f15e65eac14de4b4f870c67f69d20b2e144fe69b" gracePeriod=2
Sep 15 12:45:13.948342 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:13.948296    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7707-529c28" podStartSLOduration=1.948280858 podStartE2EDuration="1.948280858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:45:13.946024743 +0000 UTC m=+1185.330735626" watchObservedRunningTime="2026-09-15 12:45:13.948280858 +0000 UTC m=+1185.332991754"
Sep 15 12:45:14.104526 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.104502    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-aa13a3"
Sep 15 12:45:14.180670 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.180591    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs5md\" (UniqueName: \"kubernetes.io/projected/8616e403-25c9-4b0e-bed8-3c15fe2a9908-kube-api-access-hs5md\") pod \"8616e403-25c9-4b0e-bed8-3c15fe2a9908\" (UID: \"8616e403-25c9-4b0e-bed8-3c15fe2a9908\") "
Sep 15 12:45:14.182419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.182392    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8616e403-25c9-4b0e-bed8-3c15fe2a9908-kube-api-access-hs5md" (OuterVolumeSpecName: "kube-api-access-hs5md") pod "8616e403-25c9-4b0e-bed8-3c15fe2a9908" (UID: "8616e403-25c9-4b0e-bed8-3c15fe2a9908"). InnerVolumeSpecName "kube-api-access-hs5md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:45:14.281624 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.281594    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs5md\" (UniqueName: \"kubernetes.io/projected/8616e403-25c9-4b0e-bed8-3c15fe2a9908-kube-api-access-hs5md\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:45:14.508231 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.508149    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7707-529c28"]
Sep 15 12:45:14.512297 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.512265    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7707-529c28"]
Sep 15 12:45:14.624467 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.624433    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7707-2d9df7"]
Sep 15 12:45:14.624874 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.624859    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d594c6-3fe1-4b2d-854e-4f411d7eafb1" containerName="test-curl-7707-529c28"
Sep 15 12:45:14.624918 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.624878    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d594c6-3fe1-4b2d-854e-4f411d7eafb1" containerName="test-curl-7707-529c28"
Sep 15 12:45:14.624962 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.624951    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d594c6-3fe1-4b2d-854e-4f411d7eafb1" containerName="test-curl-7707-529c28"
Sep 15 12:45:14.628277 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.628251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-2d9df7"
Sep 15 12:45:14.632533 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.632504    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-2d9df7"]
Sep 15 12:45:14.686404 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.686367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxs68\" (UniqueName: \"kubernetes.io/projected/8d5b2153-feda-4cc8-bfdc-22c987842274-kube-api-access-zxs68\") pod \"test-curl-7707-2d9df7\" (UID: \"8d5b2153-feda-4cc8-bfdc-22c987842274\") " pod="opendatahub/test-curl-7707-2d9df7"
Sep 15 12:45:14.787221 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.787132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxs68\" (UniqueName: \"kubernetes.io/projected/8d5b2153-feda-4cc8-bfdc-22c987842274-kube-api-access-zxs68\") pod \"test-curl-7707-2d9df7\" (UID: \"8d5b2153-feda-4cc8-bfdc-22c987842274\") " pod="opendatahub/test-curl-7707-2d9df7"
Sep 15 12:45:14.794998 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.794972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxs68\" (UniqueName: \"kubernetes.io/projected/8d5b2153-feda-4cc8-bfdc-22c987842274-kube-api-access-zxs68\") pod \"test-curl-7707-2d9df7\" (UID: \"8d5b2153-feda-4cc8-bfdc-22c987842274\") " pod="opendatahub/test-curl-7707-2d9df7"
Sep 15 12:45:14.938744 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.938717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-2d9df7"
Sep 15 12:45:14.940234 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.940207    2579 generic.go:356] "Generic (PLEG): container finished" podID="8616e403-25c9-4b0e-bed8-3c15fe2a9908" containerID="44cce4f7e77bbfbe72e72d0aa4816503bad87879908441e9a07aa767c7bbf4ed" exitCode=137
Sep 15 12:45:14.940360 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.940261    2579 scope.go:117] "RemoveContainer" containerID="44cce4f7e77bbfbe72e72d0aa4816503bad87879908441e9a07aa767c7bbf4ed"
Sep 15 12:45:14.940360 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.940268    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-aa13a3"
Sep 15 12:45:14.940466 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:45:14.940429    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7707-529c28" container="test-curl-7707-529c28"
Sep 15 12:45:14.950133 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.950111    2579 scope.go:117] "RemoveContainer" containerID="44cce4f7e77bbfbe72e72d0aa4816503bad87879908441e9a07aa767c7bbf4ed"
Sep 15 12:45:14.950456 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:45:14.950434    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44cce4f7e77bbfbe72e72d0aa4816503bad87879908441e9a07aa767c7bbf4ed\": container with ID starting with 44cce4f7e77bbfbe72e72d0aa4816503bad87879908441e9a07aa767c7bbf4ed not found: ID does not exist" containerID="44cce4f7e77bbfbe72e72d0aa4816503bad87879908441e9a07aa767c7bbf4ed"
Sep 15 12:45:14.950516 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:14.950468    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44cce4f7e77bbfbe72e72d0aa4816503bad87879908441e9a07aa767c7bbf4ed"} err="failed to get container status \"44cce4f7e77bbfbe72e72d0aa4816503bad87879908441e9a07aa767c7bbf4ed\": rpc error: code = NotFound desc = could not find container \"44cce4f7e77bbfbe72e72d0aa4816503bad87879908441e9a07aa767c7bbf4ed\": container with ID starting with 44cce4f7e77bbfbe72e72d0aa4816503bad87879908441e9a07aa767c7bbf4ed not found: ID does not exist"
Sep 15 12:45:14.951075 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:45:14.951053    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7707-aa13a3\" is forbidden: User \"system:node:ip-10-0-129-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-17.ec2.internal' and this object" podUID="8616e403-25c9-4b0e-bed8-3c15fe2a9908" pod="opendatahub/test-curl-7707-aa13a3"
Sep 15 12:45:15.105933 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:15.105900    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8616e403-25c9-4b0e-bed8-3c15fe2a9908" path="/var/lib/kubelet/pods/8616e403-25c9-4b0e-bed8-3c15fe2a9908/volumes"
Sep 15 12:45:15.211261 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:15.211233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-2d9df7"]
Sep 15 12:45:15.212488 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:45:15.212460    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d5b2153_feda_4cc8_bfdc_22c987842274.slice/crio-d2348aa37aa589a585812add228b0d8ccbb6a4447297e800d453d189fd4579d0 WatchSource:0}: Error finding container d2348aa37aa589a585812add228b0d8ccbb6a4447297e800d453d189fd4579d0: Status 404 returned error can't find the container with id d2348aa37aa589a585812add228b0d8ccbb6a4447297e800d453d189fd4579d0
Sep 15 12:45:15.949896 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:15.949867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-2d9df7" event={"ID":"8d5b2153-feda-4cc8-bfdc-22c987842274","Type":"ContainerStarted","Data":"4199d92f89e0c224593e190168e8b6220ea80fd053cb9f92a6ac49b701e7dc63"}
Sep 15 12:45:15.950227 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:15.949909    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-2d9df7" event={"ID":"8d5b2153-feda-4cc8-bfdc-22c987842274","Type":"ContainerStarted","Data":"d2348aa37aa589a585812add228b0d8ccbb6a4447297e800d453d189fd4579d0"}
Sep 15 12:45:15.950227 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:15.950002    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7707-529c28" podUID="c2d594c6-3fe1-4b2d-854e-4f411d7eafb1" containerName="test-curl-7707-529c28" containerID="cri-o://f44c0cf4a0df89e13828c58ae02edc35176e98d685bc85a18d601a08f6ebf17e" gracePeriod=2
Sep 15 12:45:15.965525 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:15.965485    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7707-2d9df7" podStartSLOduration=1.965472422 podStartE2EDuration="1.965472422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:45:15.962933266 +0000 UTC m=+1187.347644149" watchObservedRunningTime="2026-09-15 12:45:15.965472422 +0000 UTC m=+1187.350183305"
Sep 15 12:45:16.114992 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:16.114969    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-453692"
Sep 15 12:45:16.202485 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:16.202403    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n56z\" (UniqueName: \"kubernetes.io/projected/81a05de4-8288-479e-b26a-9157f27df1b9-kube-api-access-8n56z\") pod \"81a05de4-8288-479e-b26a-9157f27df1b9\" (UID: \"81a05de4-8288-479e-b26a-9157f27df1b9\") "
Sep 15 12:45:16.204364 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:16.204341    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a05de4-8288-479e-b26a-9157f27df1b9-kube-api-access-8n56z" (OuterVolumeSpecName: "kube-api-access-8n56z") pod "81a05de4-8288-479e-b26a-9157f27df1b9" (UID: "81a05de4-8288-479e-b26a-9157f27df1b9"). InnerVolumeSpecName "kube-api-access-8n56z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:45:16.303545 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:16.303510    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n56z\" (UniqueName: \"kubernetes.io/projected/81a05de4-8288-479e-b26a-9157f27df1b9-kube-api-access-8n56z\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:45:16.535444 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:16.535361    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7707-2d9df7"]
Sep 15 12:45:16.542231 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:16.542203    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7707-2d9df7"]
Sep 15 12:45:16.956940 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:16.956904    2579 generic.go:356] "Generic (PLEG): container finished" podID="81a05de4-8288-479e-b26a-9157f27df1b9" containerID="5c495be0b5b6cdb38de82e44f15e65eac14de4b4f870c67f69d20b2e144fe69b" exitCode=137
Sep 15 12:45:16.957420 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:16.956945    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-453692"
Sep 15 12:45:16.957420 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:16.956967    2579 scope.go:117] "RemoveContainer" containerID="5c495be0b5b6cdb38de82e44f15e65eac14de4b4f870c67f69d20b2e144fe69b"
Sep 15 12:45:16.957420 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:45:16.957280    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7707-2d9df7" container="test-curl-7707-2d9df7"
Sep 15 12:45:16.966874 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:16.966856    2579 scope.go:117] "RemoveContainer" containerID="5c495be0b5b6cdb38de82e44f15e65eac14de4b4f870c67f69d20b2e144fe69b"
Sep 15 12:45:16.967145 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:45:16.967124    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c495be0b5b6cdb38de82e44f15e65eac14de4b4f870c67f69d20b2e144fe69b\": container with ID starting with 5c495be0b5b6cdb38de82e44f15e65eac14de4b4f870c67f69d20b2e144fe69b not found: ID does not exist" containerID="5c495be0b5b6cdb38de82e44f15e65eac14de4b4f870c67f69d20b2e144fe69b"
Sep 15 12:45:16.967192 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:16.967156    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c495be0b5b6cdb38de82e44f15e65eac14de4b4f870c67f69d20b2e144fe69b"} err="failed to get container status \"5c495be0b5b6cdb38de82e44f15e65eac14de4b4f870c67f69d20b2e144fe69b\": rpc error: code = NotFound desc = could not find container \"5c495be0b5b6cdb38de82e44f15e65eac14de4b4f870c67f69d20b2e144fe69b\": container with ID starting with 5c495be0b5b6cdb38de82e44f15e65eac14de4b4f870c67f69d20b2e144fe69b not found: ID does not exist"
Sep 15 12:45:16.967933 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:45:16.967912    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7707-453692\" is forbidden: User \"system:node:ip-10-0-129-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-17.ec2.internal' and this object" podUID="81a05de4-8288-479e-b26a-9157f27df1b9" pod="opendatahub/test-curl-7707-453692"
Sep 15 12:45:17.105522 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:17.105491    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81a05de4-8288-479e-b26a-9157f27df1b9" path="/var/lib/kubelet/pods/81a05de4-8288-479e-b26a-9157f27df1b9/volumes"
Sep 15 12:45:17.106980 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:17.106950    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:17.401288 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:17.401222    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:17.710331 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:17.710236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:17.963970 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:17.963941    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7707-2d9df7" podUID="8d5b2153-feda-4cc8-bfdc-22c987842274" containerName="test-curl-7707-2d9df7" containerID="cri-o://4199d92f89e0c224593e190168e8b6220ea80fd053cb9f92a6ac49b701e7dc63" gracePeriod=2
Sep 15 12:45:18.119295 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:18.119272    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-529c28"
Sep 15 12:45:18.221997 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:18.221911    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brmhb\" (UniqueName: \"kubernetes.io/projected/c2d594c6-3fe1-4b2d-854e-4f411d7eafb1-kube-api-access-brmhb\") pod \"c2d594c6-3fe1-4b2d-854e-4f411d7eafb1\" (UID: \"c2d594c6-3fe1-4b2d-854e-4f411d7eafb1\") "
Sep 15 12:45:18.224018 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:18.223988    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d594c6-3fe1-4b2d-854e-4f411d7eafb1-kube-api-access-brmhb" (OuterVolumeSpecName: "kube-api-access-brmhb") pod "c2d594c6-3fe1-4b2d-854e-4f411d7eafb1" (UID: "c2d594c6-3fe1-4b2d-854e-4f411d7eafb1"). InnerVolumeSpecName "kube-api-access-brmhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:45:18.323627 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:18.323597    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brmhb\" (UniqueName: \"kubernetes.io/projected/c2d594c6-3fe1-4b2d-854e-4f411d7eafb1-kube-api-access-brmhb\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:45:18.973360 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:18.973320    2579 generic.go:356] "Generic (PLEG): container finished" podID="c2d594c6-3fe1-4b2d-854e-4f411d7eafb1" containerID="f44c0cf4a0df89e13828c58ae02edc35176e98d685bc85a18d601a08f6ebf17e" exitCode=137
Sep 15 12:45:18.973360 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:18.973357    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-529c28"
Sep 15 12:45:18.973852 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:18.973394    2579 scope.go:117] "RemoveContainer" containerID="f44c0cf4a0df89e13828c58ae02edc35176e98d685bc85a18d601a08f6ebf17e"
Sep 15 12:45:18.982089 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:18.982072    2579 scope.go:117] "RemoveContainer" containerID="f44c0cf4a0df89e13828c58ae02edc35176e98d685bc85a18d601a08f6ebf17e"
Sep 15 12:45:18.988589 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:45:18.988548    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44c0cf4a0df89e13828c58ae02edc35176e98d685bc85a18d601a08f6ebf17e\": container with ID starting with f44c0cf4a0df89e13828c58ae02edc35176e98d685bc85a18d601a08f6ebf17e not found: ID does not exist" containerID="f44c0cf4a0df89e13828c58ae02edc35176e98d685bc85a18d601a08f6ebf17e"
Sep 15 12:45:18.988702 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:18.988600    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44c0cf4a0df89e13828c58ae02edc35176e98d685bc85a18d601a08f6ebf17e"} err="failed to get container status \"f44c0cf4a0df89e13828c58ae02edc35176e98d685bc85a18d601a08f6ebf17e\": rpc error: code = NotFound desc = could not find container \"f44c0cf4a0df89e13828c58ae02edc35176e98d685bc85a18d601a08f6ebf17e\": container with ID starting with f44c0cf4a0df89e13828c58ae02edc35176e98d685bc85a18d601a08f6ebf17e not found: ID does not exist"
Sep 15 12:45:18.988702 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:45:18.988678    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7707-529c28\" is forbidden: User \"system:node:ip-10-0-129-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-17.ec2.internal' and this object" podUID="c2d594c6-3fe1-4b2d-854e-4f411d7eafb1" pod="opendatahub/test-curl-7707-529c28"
Sep 15 12:45:19.104435 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:45:19.104391    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7707-529c28\" is forbidden: User \"system:node:ip-10-0-129-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-17.ec2.internal' and this object" podUID="c2d594c6-3fe1-4b2d-854e-4f411d7eafb1" pod="opendatahub/test-curl-7707-529c28"
Sep 15 12:45:19.105504 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:19.105483    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2d594c6-3fe1-4b2d-854e-4f411d7eafb1" path="/var/lib/kubelet/pods/c2d594c6-3fe1-4b2d-854e-4f411d7eafb1/volumes"
Sep 15 12:45:20.134557 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:20.134530    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-2d9df7"
Sep 15 12:45:20.200913 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:20.200865    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:20.243137 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:20.243068    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxs68\" (UniqueName: \"kubernetes.io/projected/8d5b2153-feda-4cc8-bfdc-22c987842274-kube-api-access-zxs68\") pod \"8d5b2153-feda-4cc8-bfdc-22c987842274\" (UID: \"8d5b2153-feda-4cc8-bfdc-22c987842274\") "
Sep 15 12:45:20.245139 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:20.245110    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5b2153-feda-4cc8-bfdc-22c987842274-kube-api-access-zxs68" (OuterVolumeSpecName: "kube-api-access-zxs68") pod "8d5b2153-feda-4cc8-bfdc-22c987842274" (UID: "8d5b2153-feda-4cc8-bfdc-22c987842274"). InnerVolumeSpecName "kube-api-access-zxs68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:45:20.344415 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:20.344381    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxs68\" (UniqueName: \"kubernetes.io/projected/8d5b2153-feda-4cc8-bfdc-22c987842274-kube-api-access-zxs68\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:45:20.800612 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:20.800576    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:20.892470 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:20.892428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:20.987235 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:20.987199    2579 generic.go:356] "Generic (PLEG): container finished" podID="8d5b2153-feda-4cc8-bfdc-22c987842274" containerID="4199d92f89e0c224593e190168e8b6220ea80fd053cb9f92a6ac49b701e7dc63" exitCode=137
Sep 15 12:45:20.987405 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:20.987249    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-2d9df7"
Sep 15 12:45:20.987405 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:20.987299    2579 scope.go:117] "RemoveContainer" containerID="4199d92f89e0c224593e190168e8b6220ea80fd053cb9f92a6ac49b701e7dc63"
Sep 15 12:45:20.996574 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:20.996389    2579 scope.go:117] "RemoveContainer" containerID="4199d92f89e0c224593e190168e8b6220ea80fd053cb9f92a6ac49b701e7dc63"
Sep 15 12:45:20.996686 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:45:20.996668    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4199d92f89e0c224593e190168e8b6220ea80fd053cb9f92a6ac49b701e7dc63\": container with ID starting with 4199d92f89e0c224593e190168e8b6220ea80fd053cb9f92a6ac49b701e7dc63 not found: ID does not exist" containerID="4199d92f89e0c224593e190168e8b6220ea80fd053cb9f92a6ac49b701e7dc63"
Sep 15 12:45:20.996732 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:20.996693    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4199d92f89e0c224593e190168e8b6220ea80fd053cb9f92a6ac49b701e7dc63"} err="failed to get container status \"4199d92f89e0c224593e190168e8b6220ea80fd053cb9f92a6ac49b701e7dc63\": rpc error: code = NotFound desc = could not find container \"4199d92f89e0c224593e190168e8b6220ea80fd053cb9f92a6ac49b701e7dc63\": container with ID starting with 4199d92f89e0c224593e190168e8b6220ea80fd053cb9f92a6ac49b701e7dc63 not found: ID does not exist"
Sep 15 12:45:20.997525 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:45:20.997501    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7707-2d9df7\" is forbidden: User \"system:node:ip-10-0-129-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-17.ec2.internal' and this object" podUID="8d5b2153-feda-4cc8-bfdc-22c987842274" pod="opendatahub/test-curl-7707-2d9df7"
Sep 15 12:45:21.106056 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:21.106026    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d5b2153-feda-4cc8-bfdc-22c987842274" path="/var/lib/kubelet/pods/8d5b2153-feda-4cc8-bfdc-22c987842274/volumes"
Sep 15 12:45:21.179006 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:21.178976    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:21.993100 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:21.993062    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:22.679108 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:22.679069    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:23.477162 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:23.477128    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:24.274841 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:24.274785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:24.878372 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:24.878337    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:25.567587 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.567550    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"]
Sep 15 12:45:25.567994 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.567967    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d5b2153-feda-4cc8-bfdc-22c987842274" containerName="test-curl-7707-2d9df7"
Sep 15 12:45:25.567994 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.567980    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d5b2153-feda-4cc8-bfdc-22c987842274" containerName="test-curl-7707-2d9df7"
Sep 15 12:45:25.568070 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.568035    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d5b2153-feda-4cc8-bfdc-22c987842274" containerName="test-curl-7707-2d9df7"
Sep 15 12:45:25.572406 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.572387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:25.575898 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.575875    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-88615e8e-serving-cert\""
Sep 15 12:45:25.581913 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.581326    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"]
Sep 15 12:45:25.598821 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.598796    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:25.695318 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.695281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5xdw\" (UniqueName: \"kubernetes.io/projected/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-kube-api-access-l5xdw\") pod \"maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:25.695318 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.695326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-openshift-service-ca\") pod \"maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:25.695574 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.695353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-maas-api-tls\") pod \"maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:25.695574 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.695465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-metrics-certs\") pod \"maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:25.700588 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.700556    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:25.796757 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.796717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-metrics-certs\") pod \"maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:25.796962 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.796846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5xdw\" (UniqueName: \"kubernetes.io/projected/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-kube-api-access-l5xdw\") pod \"maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:25.796962 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.796882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-openshift-service-ca\") pod \"maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:25.797098 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.797011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-maas-api-tls\") pod \"maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:25.797161 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:45:25.797124    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-serving-cert: secret "maas-api-e2e-mt-88615e8e-serving-cert" not found
Sep 15 12:45:25.797219 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:45:25.797201    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-maas-api-tls podName:fa0cd320-b68d-49f8-bd29-fcfb91ba467f nodeName:}" failed. No retries permitted until 2026-09-15 12:45:26.297184363 +0000 UTC m=+1197.681895225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-maas-api-tls") pod "maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7" (UID: "fa0cd320-b68d-49f8-bd29-fcfb91ba467f") : secret "maas-api-e2e-mt-88615e8e-serving-cert" not found
Sep 15 12:45:25.797582 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.797560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-openshift-service-ca\") pod \"maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:25.799184 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.799168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-metrics-certs\") pod \"maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:25.805114 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.805096    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5xdw\" (UniqueName: \"kubernetes.io/projected/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-kube-api-access-l5xdw\") pod \"maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:25.867982 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:25.867946    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:26.062692 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:26.062656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:26.302269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:26.302179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-maas-api-tls\") pod \"maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:26.304607 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:26.304583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-maas-api-tls\") pod \"maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:26.486475 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:26.486439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:26.758661 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:26.758635    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"]
Sep 15 12:45:26.760258 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:45:26.760234    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0cd320_b68d_49f8_bd29_fcfb91ba467f.slice/crio-06f8a0fb83518ed2d25d3fc0f2080590f781aa4c0e9292c1048b4f8f2b807227 WatchSource:0}: Error finding container 06f8a0fb83518ed2d25d3fc0f2080590f781aa4c0e9292c1048b4f8f2b807227: Status 404 returned error can't find the container with id 06f8a0fb83518ed2d25d3fc0f2080590f781aa4c0e9292c1048b4f8f2b807227
Sep 15 12:45:27.022886 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:27.022782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7" event={"ID":"fa0cd320-b68d-49f8-bd29-fcfb91ba467f","Type":"ContainerStarted","Data":"06f8a0fb83518ed2d25d3fc0f2080590f781aa4c0e9292c1048b4f8f2b807227"}
Sep 15 12:45:27.058689 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:27.058654    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:28.030042 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:28.030008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7" event={"ID":"fa0cd320-b68d-49f8-bd29-fcfb91ba467f","Type":"ContainerStarted","Data":"d155f4496243ef5e679c3642b76f40350752046509c5401c0643b683d765a6b1"}
Sep 15 12:45:28.030545 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:28.030115    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:28.047710 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:28.047658    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7" podStartSLOduration=2.671776223 podStartE2EDuration="3.047640464s" totalImagesPullingTime="375.864241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:45:28.04649349 +0000 UTC m=+1199.431204377" watchObservedRunningTime="2026-09-15 12:45:28.047640464 +0000 UTC m=+1199.432351345"
Sep 15 12:45:29.177040 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:29.177004    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:45:29.178966 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:29.178940    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:45:34.040325 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:34.040297    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:45:35.865891 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:35.865853    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:45:43.340894 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:43.340858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:43.567163 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:43.567121    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:43.677487 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:43.677452    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:47.787896 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:47.787861    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:48.734273 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:48.734242    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:48.794953 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:48.794911    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:49.535418 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:49.535375    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:51.679250 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:51.679215    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:51.774621 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:51.774588    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:52.613007 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:52.612971    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:53.397027 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:53.396984    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:54.379336 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:54.379306    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:55.248572 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.248537    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"]
Sep 15 12:45:55.252188 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.252170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.254777 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.254752    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-11f865\"/\"distinct-w0-11f865-kserve-self-signed-certs\""
Sep 15 12:45:55.260954 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.260932    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"]
Sep 15 12:45:55.276351 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.276321    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:55.286547 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.286528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-dshm\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.286609 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.286570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-model-cache\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.286609 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.286593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-tls-certs\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.286713 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.286697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-tmp-dir\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.286750 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.286729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-home\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.286788 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.286752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnzdl\" (UniqueName: \"kubernetes.io/projected/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-kube-api-access-qnzdl\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.387229 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.387197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-tmp-dir\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.387229 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.387233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-home\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.387437 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.387256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnzdl\" (UniqueName: \"kubernetes.io/projected/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-kube-api-access-qnzdl\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.387437 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.387294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-dshm\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.387437 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.387349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-model-cache\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.387437 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.387380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-tls-certs\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.387676 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.387657    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-tmp-dir\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.387740 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.387688    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-home\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.387740 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.387710    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-model-cache\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.389577 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.389559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-dshm\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.389879 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.389862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-tls-certs\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.394925 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.394903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnzdl\" (UniqueName: \"kubernetes.io/projected/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-kube-api-access-qnzdl\") pod \"distinct-w0-11f865-kserve-586d469ddb-pbd6q\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.563030 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.562925    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:45:55.834482 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:55.834455    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"]
Sep 15 12:45:55.836898 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:45:55.836866    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44ccd6bb_d7db_461f_be67_a6f7312cb5b1.slice/crio-8de17bb73211c32a3e24ba1420b97c82e4a7cf1736acc7856390e2e316eadfd6 WatchSource:0}: Error finding container 8de17bb73211c32a3e24ba1420b97c82e4a7cf1736acc7856390e2e316eadfd6: Status 404 returned error can't find the container with id 8de17bb73211c32a3e24ba1420b97c82e4a7cf1736acc7856390e2e316eadfd6
Sep 15 12:45:56.182337 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:56.182300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q" event={"ID":"44ccd6bb-d7db-461f-be67-a6f7312cb5b1","Type":"ContainerStarted","Data":"1c9bd9af00fb502c2976eb6e2be14d1c0f6d592264ddcc691cb300048f88bca1"}
Sep 15 12:45:56.182507 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:56.182345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q" event={"ID":"44ccd6bb-d7db-461f-be67-a6f7312cb5b1","Type":"ContainerStarted","Data":"8de17bb73211c32a3e24ba1420b97c82e4a7cf1736acc7856390e2e316eadfd6"}
Sep 15 12:45:56.200973 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:56.200926    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q" podStartSLOduration=1.200910779 podStartE2EDuration="1.200910779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:45:56.199273616 +0000 UTC m=+1227.583984501" watchObservedRunningTime="2026-09-15 12:45:56.200910779 +0000 UTC m=+1227.585621697"
Sep 15 12:45:57.069531 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:57.069498    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:45:57.119218 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:45:57.119185    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:02.850254 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:02.850220    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"]
Sep 15 12:46:02.853643 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:02.853625    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:02.856284 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:02.856262    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ee96ec\"/\"e2e-conflict-b-2381886b-kserve-self-signed-certs\""
Sep 15 12:46:02.864347 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:02.864327    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"]
Sep 15 12:46:02.875184 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:02.875156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:02.959195 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:02.959162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-dshm\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:02.959361 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:02.959219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-tmp-dir\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:02.959361 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:02.959294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr8gq\" (UniqueName: \"kubernetes.io/projected/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-kube-api-access-xr8gq\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:02.959441 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:02.959366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-model-cache\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:02.959441 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:02.959432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-home\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:02.959519 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:02.959447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-tls-certs\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:03.059951 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:03.059921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-dshm\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:03.060152 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:03.059972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-tmp-dir\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:03.060152 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:03.059998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr8gq\" (UniqueName: \"kubernetes.io/projected/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-kube-api-access-xr8gq\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:03.060152 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:03.060027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-model-cache\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:03.060152 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:03.060071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-home\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:03.060152 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:03.060085    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-tls-certs\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:03.060414 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:03.060376    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-tmp-dir\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:03.060475 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:03.060413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-model-cache\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:03.060532 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:03.060509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-home\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:03.062283 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:03.062256    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-dshm\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:03.062777 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:03.062757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-tls-certs\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:03.067500 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:03.067475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr8gq\" (UniqueName: \"kubernetes.io/projected/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-kube-api-access-xr8gq\") pod \"e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") " pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:03.163585 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:03.163505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:03.433614 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:03.433587    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"]
Sep 15 12:46:03.435477 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:46:03.435453    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffb4c622_7abc_4313_bd77_c6ed9e0a38df.slice/crio-8ae6691e505ac1c33337b5eaa16d1324bd28afe4a26c9c4666063435887e428a WatchSource:0}: Error finding container 8ae6691e505ac1c33337b5eaa16d1324bd28afe4a26c9c4666063435887e428a: Status 404 returned error can't find the container with id 8ae6691e505ac1c33337b5eaa16d1324bd28afe4a26c9c4666063435887e428a
Sep 15 12:46:03.883812 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:03.883775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:04.226993 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:04.226901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd" event={"ID":"ffb4c622-7abc-4313-bd77-c6ed9e0a38df","Type":"ContainerStarted","Data":"bdc2602c7610366f4d70949de5db6f54f98afcc5755ac3c7b436b5a2a955c238"}
Sep 15 12:46:04.226993 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:04.226946    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd" event={"ID":"ffb4c622-7abc-4313-bd77-c6ed9e0a38df","Type":"ContainerStarted","Data":"8ae6691e505ac1c33337b5eaa16d1324bd28afe4a26c9c4666063435887e428a"}
Sep 15 12:46:04.242484 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:04.242435    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd" podStartSLOduration=2.24242063 podStartE2EDuration="2.24242063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:46:04.241573508 +0000 UTC m=+1235.626284405" watchObservedRunningTime="2026-09-15 12:46:04.24242063 +0000 UTC m=+1235.627131515"
Sep 15 12:46:05.563918 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:05.563868    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:46:05.563918 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:05.563925    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:46:05.576338 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:05.576310    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:46:06.243607 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:06.243564    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:06.250382 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:06.250360    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:46:06.931328 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:06.931295    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"]
Sep 15 12:46:06.949629 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:06.949597    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:08.187871 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:08.187811    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:08.593978 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:08.593940    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:09.399754 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:09.399718    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:10.069772 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:10.069738    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:10.879914 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:10.879877    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:11.535332 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:11.535298    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:12.193636 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:12.193598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:12.493916 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:12.493823    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:46:13.163797 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:13.163761    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:13.164003 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:13.163811    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:13.176813 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:13.176783    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:13.205714 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:13.205678    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:13.289195 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:13.289165    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:46:13.913187 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:13.913153    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:14.696755 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:14.696717    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:18.809156 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:18.809116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:18.880327 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:18.880292    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:18.940644 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:18.940608    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:19.002740 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:19.002701    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:19.063561 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:19.063450    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:19.363354 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:19.363324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:19.464949 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:19.464918    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:20.282546 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:20.282513    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:21.115371 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:21.115330    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:21.789899 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:21.789867    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:21.951636 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:21.951597    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76" podUID="7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3" containerName="main" containerID="cri-o://644873ecee75a0a02b676a6b1d3bfd99b9c6199e62181d8d7069bf71b2bacb47" gracePeriod=45
Sep 15 12:46:22.194778 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.194753    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:46:22.334522 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.334435    2579 generic.go:356] "Generic (PLEG): container finished" podID="7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3" containerID="644873ecee75a0a02b676a6b1d3bfd99b9c6199e62181d8d7069bf71b2bacb47" exitCode=0
Sep 15 12:46:22.334522 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.334491    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"
Sep 15 12:46:22.334719 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.334520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76" event={"ID":"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3","Type":"ContainerDied","Data":"644873ecee75a0a02b676a6b1d3bfd99b9c6199e62181d8d7069bf71b2bacb47"}
Sep 15 12:46:22.334719 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.334563    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76" event={"ID":"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3","Type":"ContainerDied","Data":"f0df293d56d732763f9ea5c66b612dc5d7f974a51e89fc8f1621da47a1ac11fe"}
Sep 15 12:46:22.334719 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.334579    2579 scope.go:117] "RemoveContainer" containerID="644873ecee75a0a02b676a6b1d3bfd99b9c6199e62181d8d7069bf71b2bacb47"
Sep 15 12:46:22.343503 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.343483    2579 scope.go:117] "RemoveContainer" containerID="644873ecee75a0a02b676a6b1d3bfd99b9c6199e62181d8d7069bf71b2bacb47"
Sep 15 12:46:22.343796 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:46:22.343778    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644873ecee75a0a02b676a6b1d3bfd99b9c6199e62181d8d7069bf71b2bacb47\": container with ID starting with 644873ecee75a0a02b676a6b1d3bfd99b9c6199e62181d8d7069bf71b2bacb47 not found: ID does not exist" containerID="644873ecee75a0a02b676a6b1d3bfd99b9c6199e62181d8d7069bf71b2bacb47"
Sep 15 12:46:22.343869 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.343804    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644873ecee75a0a02b676a6b1d3bfd99b9c6199e62181d8d7069bf71b2bacb47"} err="failed to get container status \"644873ecee75a0a02b676a6b1d3bfd99b9c6199e62181d8d7069bf71b2bacb47\": rpc error: code = NotFound desc = could not find container \"644873ecee75a0a02b676a6b1d3bfd99b9c6199e62181d8d7069bf71b2bacb47\": container with ID starting with 644873ecee75a0a02b676a6b1d3bfd99b9c6199e62181d8d7069bf71b2bacb47 not found: ID does not exist"
Sep 15 12:46:22.353151 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.353133    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-dshm\") pod \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") "
Sep 15 12:46:22.353240 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.353181    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-home\") pod \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") "
Sep 15 12:46:22.353240 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.353209    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-model-cache\") pod \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") "
Sep 15 12:46:22.353339 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.353265    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-tmp-dir\") pod \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") "
Sep 15 12:46:22.353339 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.353296    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-tls-certs\") pod \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") "
Sep 15 12:46:22.353449 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.353357    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtcvm\" (UniqueName: \"kubernetes.io/projected/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-kube-api-access-rtcvm\") pod \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\" (UID: \"7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3\") "
Sep 15 12:46:22.353449 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.353420    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-home" (OuterVolumeSpecName: "home") pod "7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3" (UID: "7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:46:22.353559 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.353471    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3" (UID: "7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:46:22.353615 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.353575    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3" (UID: "7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:46:22.353681 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.353663    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:46:22.353726 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.353687    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:46:22.353726 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.353703    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:46:22.355403 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.355371    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3" (UID: "7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:46:22.355512 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.355421    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-kube-api-access-rtcvm" (OuterVolumeSpecName: "kube-api-access-rtcvm") pod "7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3" (UID: "7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3"). InnerVolumeSpecName "kube-api-access-rtcvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:46:22.355512 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.355432    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-dshm" (OuterVolumeSpecName: "dshm") pod "7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3" (UID: "7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:46:22.454765 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.454723    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:46:22.454765 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.454755    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:46:22.454765 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.454766    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtcvm\" (UniqueName: \"kubernetes.io/projected/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3-kube-api-access-rtcvm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:46:22.584735 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.584623    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:22.656650 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.656617    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"]
Sep 15 12:46:22.662267 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:22.662241    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4609ba/test-model-4609ba-kserve-649698b8d9-44j76"]
Sep 15 12:46:23.106351 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:23.106319    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3" path="/var/lib/kubelet/pods/7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3/volumes"
Sep 15 12:46:23.227690 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:23.227662    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:23.876952 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:23.876917    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:25.338270 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:25.338229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:25.430281 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:25.430245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:26.630685 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:26.630646    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:26.733254 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:26.733202    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:26.898037 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:26.897931    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:27.621970 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:27.621938    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:28.285896 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:28.285864    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:28.986195 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:28.986162    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:29.653981 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:29.653945    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:30.455450 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:30.455415    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:31.209010 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:31.208972    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:31.478069 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:31.477988    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:36.366541 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.366504    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"]
Sep 15 12:46:36.393958 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.393910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:36.394907 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.394878    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e3fef-29824605-bjzr6"]
Sep 15 12:46:36.397296 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.397271    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2e3fef-29824605-bjzr6"]
Sep 15 12:46:36.415439 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.415405    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw" podUID="66a5635c-5286-47e9-8270-5e82916b4156" containerName="maas-api" containerID="cri-o://7dacc829cd19c0218d13f7b2db564334e0ff30747aeccf2042a04aff8502eeb6" gracePeriod=30
Sep 15 12:46:36.460513 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.460484    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:36.663569 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.663549    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:46:36.689671 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.689647    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66a5635c-5286-47e9-8270-5e82916b4156-openshift-service-ca\") pod \"66a5635c-5286-47e9-8270-5e82916b4156\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") "
Sep 15 12:46:36.689802 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.689704    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66a5635c-5286-47e9-8270-5e82916b4156-maas-api-tls\") pod \"66a5635c-5286-47e9-8270-5e82916b4156\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") "
Sep 15 12:46:36.689802 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.689774    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a5635c-5286-47e9-8270-5e82916b4156-metrics-certs\") pod \"66a5635c-5286-47e9-8270-5e82916b4156\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") "
Sep 15 12:46:36.689911 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.689817    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n7xz\" (UniqueName: \"kubernetes.io/projected/66a5635c-5286-47e9-8270-5e82916b4156-kube-api-access-6n7xz\") pod \"66a5635c-5286-47e9-8270-5e82916b4156\" (UID: \"66a5635c-5286-47e9-8270-5e82916b4156\") "
Sep 15 12:46:36.690131 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.690102    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a5635c-5286-47e9-8270-5e82916b4156-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66a5635c-5286-47e9-8270-5e82916b4156" (UID: "66a5635c-5286-47e9-8270-5e82916b4156"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:46:36.692001 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.691970    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a5635c-5286-47e9-8270-5e82916b4156-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "66a5635c-5286-47e9-8270-5e82916b4156" (UID: "66a5635c-5286-47e9-8270-5e82916b4156"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:46:36.692098 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.691995    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a5635c-5286-47e9-8270-5e82916b4156-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66a5635c-5286-47e9-8270-5e82916b4156" (UID: "66a5635c-5286-47e9-8270-5e82916b4156"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:46:36.692098 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.692038    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a5635c-5286-47e9-8270-5e82916b4156-kube-api-access-6n7xz" (OuterVolumeSpecName: "kube-api-access-6n7xz") pod "66a5635c-5286-47e9-8270-5e82916b4156" (UID: "66a5635c-5286-47e9-8270-5e82916b4156"). InnerVolumeSpecName "kube-api-access-6n7xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:46:36.790818 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.790787    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a5635c-5286-47e9-8270-5e82916b4156-metrics-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:46:36.790818 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.790814    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n7xz\" (UniqueName: \"kubernetes.io/projected/66a5635c-5286-47e9-8270-5e82916b4156-kube-api-access-6n7xz\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:46:36.791014 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.790823    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66a5635c-5286-47e9-8270-5e82916b4156-openshift-service-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:46:36.791014 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:36.790857    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66a5635c-5286-47e9-8270-5e82916b4156-maas-api-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:46:37.105441 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:37.105409    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c787bc6-85d8-42f9-9e75-332a69028fa4" path="/var/lib/kubelet/pods/9c787bc6-85d8-42f9-9e75-332a69028fa4/volumes"
Sep 15 12:46:37.422755 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:37.422670    2579 generic.go:356] "Generic (PLEG): container finished" podID="66a5635c-5286-47e9-8270-5e82916b4156" containerID="7dacc829cd19c0218d13f7b2db564334e0ff30747aeccf2042a04aff8502eeb6" exitCode=0
Sep 15 12:46:37.422755 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:37.422720    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"
Sep 15 12:46:37.423229 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:37.422760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw" event={"ID":"66a5635c-5286-47e9-8270-5e82916b4156","Type":"ContainerDied","Data":"7dacc829cd19c0218d13f7b2db564334e0ff30747aeccf2042a04aff8502eeb6"}
Sep 15 12:46:37.423229 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:37.422819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw" event={"ID":"66a5635c-5286-47e9-8270-5e82916b4156","Type":"ContainerDied","Data":"66236b19ff08a9c583f35a2508e513039c448e7e3f80493c48ae95ddc50cc771"}
Sep 15 12:46:37.423229 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:37.422861    2579 scope.go:117] "RemoveContainer" containerID="7dacc829cd19c0218d13f7b2db564334e0ff30747aeccf2042a04aff8502eeb6"
Sep 15 12:46:37.433817 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:37.433799    2579 scope.go:117] "RemoveContainer" containerID="7dacc829cd19c0218d13f7b2db564334e0ff30747aeccf2042a04aff8502eeb6"
Sep 15 12:46:37.434116 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:46:37.434097    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dacc829cd19c0218d13f7b2db564334e0ff30747aeccf2042a04aff8502eeb6\": container with ID starting with 7dacc829cd19c0218d13f7b2db564334e0ff30747aeccf2042a04aff8502eeb6 not found: ID does not exist" containerID="7dacc829cd19c0218d13f7b2db564334e0ff30747aeccf2042a04aff8502eeb6"
Sep 15 12:46:37.434170 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:37.434124    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dacc829cd19c0218d13f7b2db564334e0ff30747aeccf2042a04aff8502eeb6"} err="failed to get container status \"7dacc829cd19c0218d13f7b2db564334e0ff30747aeccf2042a04aff8502eeb6\": rpc error: code = NotFound desc = could not find container \"7dacc829cd19c0218d13f7b2db564334e0ff30747aeccf2042a04aff8502eeb6\": container with ID starting with 7dacc829cd19c0218d13f7b2db564334e0ff30747aeccf2042a04aff8502eeb6 not found: ID does not exist"
Sep 15 12:46:37.442394 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:37.442372    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"]
Sep 15 12:46:37.445710 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:37.445689    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2e3fef-5cf6cd777d-4bjtw"]
Sep 15 12:46:37.547877 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:37.547822    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:37.648689 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:37.648649    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:37.752603 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:37.752514    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:38.455189 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:38.455142    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:39.107073 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:39.107039    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66a5635c-5286-47e9-8270-5e82916b4156" path="/var/lib/kubelet/pods/66a5635c-5286-47e9-8270-5e82916b4156/volumes"
Sep 15 12:46:39.210780 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:39.210748    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:39.878021 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:39.877989    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:40.569487 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:40.569452    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:42.639765 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:42.639723    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:42.740392 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:42.740357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:42.899036 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:42.898946    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:43.451380 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:43.451342    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:43.756474 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:43.756387    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:44.147413 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:44.147377    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:46.302619 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:46.302587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:46.963525 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:46.963483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:47.064294 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:47.064261    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:47.912960 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:47.912924    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:48.682452 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:48.682414    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:49.409669 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:49.409636    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:50.065949 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:50.065914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:50.896254 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:50.896222    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:50.983016 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:50.982981    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:51.235770 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:51.235675    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:51.359784 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:51.359746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:51.461115 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:51.461076    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:46:53.257081 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:46:53.257043    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:47:12.777152 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:12.777111    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:13.615296 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:13.615259    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:14.294592 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:14.294562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:15.085186 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:15.085148    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:15.793855 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:15.793802    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:16.568851 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:16.568794    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:17.168085 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:17.168043    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:17.386918 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:17.386881    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:17.558793 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:17.558704    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:17.886626 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:17.886574    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:17.985974 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:17.985936    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:21.129214 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:21.129178    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:21.230256 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:21.230219    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:21.675063 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:21.675029    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:22.420335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:22.420303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:23.103663 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:23.103615    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:23.886394 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:23.886360    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:24.574249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:24.574210    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:25.188228 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.188196    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"]
Sep 15 12:47:25.188733 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.188717    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66a5635c-5286-47e9-8270-5e82916b4156" containerName="maas-api"
Sep 15 12:47:25.188780 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.188736    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="66a5635c-5286-47e9-8270-5e82916b4156" containerName="maas-api"
Sep 15 12:47:25.188780 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.188776    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3" containerName="main"
Sep 15 12:47:25.188874 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.188786    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3" containerName="main"
Sep 15 12:47:25.188912 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.188883    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="66a5635c-5286-47e9-8270-5e82916b4156" containerName="maas-api"
Sep 15 12:47:25.188912 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.188898    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e232a06-87d3-4b04-9ff1-afbe6b0cf4c3" containerName="main"
Sep 15 12:47:25.192131 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.192110    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:25.194712 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.194684    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-17f8f8-serving-cert\""
Sep 15 12:47:25.203025 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.203000    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"]
Sep 15 12:47:25.214511 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.214483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:25.257731 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.257698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:25.257899 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.257745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8dth\" (UniqueName: \"kubernetes.io/projected/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-kube-api-access-k8dth\") pod \"maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:25.257899 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.257869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:25.258029 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.257931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:25.315749 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.315714    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:25.358469 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.358436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:25.358625 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.358485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:25.358625 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.358545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:25.358625 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.358584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dth\" (UniqueName: \"kubernetes.io/projected/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-kube-api-access-k8dth\") pod \"maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:25.358743 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:47:25.358655    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-serving-cert: secret "maas-api-e2e-shared-b-w3-17f8f8-serving-cert" not found
Sep 15 12:47:25.358743 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:47:25.358735    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-maas-api-tls podName:136a2d5c-2cb7-4f35-8df6-d1eb52af7467 nodeName:}" failed. No retries permitted until 2026-09-15 12:47:25.858713436 +0000 UTC m=+1317.243424314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-maas-api-tls") pod "maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75" (UID: "136a2d5c-2cb7-4f35-8df6-d1eb52af7467") : secret "maas-api-e2e-shared-b-w3-17f8f8-serving-cert" not found
Sep 15 12:47:25.359253 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.359233    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:25.361006 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.360983    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:25.367606 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.367582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8dth\" (UniqueName: \"kubernetes.io/projected/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-kube-api-access-k8dth\") pod \"maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:25.535467 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.535374    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:25.665603 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.665570    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:47:25.683341 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.683310    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:25.771746 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.771718    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:25.864895 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.864859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:25.867194 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:25.867167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:26.105884 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:26.105849    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:26.381557 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:26.381526    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"]
Sep 15 12:47:26.382805 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:47:26.382781    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136a2d5c_2cb7_4f35_8df6_d1eb52af7467.slice/crio-36f72af98f244b04cbd04d0e652418998f1512f285f551276e7bff4ab638f40f WatchSource:0}: Error finding container 36f72af98f244b04cbd04d0e652418998f1512f285f551276e7bff4ab638f40f: Status 404 returned error can't find the container with id 36f72af98f244b04cbd04d0e652418998f1512f285f551276e7bff4ab638f40f
Sep 15 12:47:26.384238 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:26.384220    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:47:26.710066 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:26.709976    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75" event={"ID":"136a2d5c-2cb7-4f35-8df6-d1eb52af7467","Type":"ContainerStarted","Data":"36f72af98f244b04cbd04d0e652418998f1512f285f551276e7bff4ab638f40f"}
Sep 15 12:47:28.722990 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:28.722952    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75" event={"ID":"136a2d5c-2cb7-4f35-8df6-d1eb52af7467","Type":"ContainerStarted","Data":"c07b0dd6c08834da1318dd5d7232e88fced2cf56b80b4ee9e43428240a6b0cd8"}
Sep 15 12:47:28.723558 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:28.723090    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:28.740721 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:28.740676    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75" podStartSLOduration=1.9533338279999999 podStartE2EDuration="3.740661212s" totalImagesPullingTime="1.787327384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:47:28.738624816 +0000 UTC m=+1320.123335713" watchObservedRunningTime="2026-09-15 12:47:28.740661212 +0000 UTC m=+1320.125372097"
Sep 15 12:47:34.734339 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:34.734309    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 12:47:47.408562 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:47.408522    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:48.157847 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:48.157800    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:48.965775 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:48.965738    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:49.710051 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:49.710007    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:50.483144 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:50.483107    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:50.667969 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:50.667933    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:51.321601 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:51.321560    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:51.421140 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:51.421103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:52.298576 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:52.298545    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:52.981243 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:52.981215    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:55.431459 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:55.431420    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:47:55.531619 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:47:55.531581    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:00.261765 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:00.261725    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:48:18.369093 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:18.369053    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:19.364123 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:19.364088    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:19.761787 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:19.761694    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:20.501635 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:20.501592    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:21.168294 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:21.168242    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:22.273605 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:22.273574    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:22.657007 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:22.656965    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:23.402138 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:23.402080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:24.077507 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:24.077469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:24.165018 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:24.164988    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:48:25.387634 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:25.387598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:25.952776 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:25.952741    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"]
Sep 15 12:48:25.966370 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:25.966341    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"]
Sep 15 12:48:25.966546 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:25.966439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:25.969281 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:25.969257    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-11f865\"/\"distinct-2-w0-11f865-kserve-self-signed-certs\""
Sep 15 12:48:25.986533 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:25.986500    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:26.063309 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.063268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6dfs\" (UniqueName: \"kubernetes.io/projected/ef085a68-d4f8-466f-b7dc-2d181289f25e-kube-api-access-x6dfs\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.063515 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.063316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef085a68-d4f8-466f-b7dc-2d181289f25e-tls-certs\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.063515 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.063399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-model-cache\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.063515 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.063418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-tmp-dir\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.063644 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.063514    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-home\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.063644 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.063579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-dshm\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.164221 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.164186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-home\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.164400 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.164246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-dshm\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.164400 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.164271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6dfs\" (UniqueName: \"kubernetes.io/projected/ef085a68-d4f8-466f-b7dc-2d181289f25e-kube-api-access-x6dfs\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.164400 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.164296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef085a68-d4f8-466f-b7dc-2d181289f25e-tls-certs\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.164400 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.164342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-model-cache\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.164400 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.164366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-tmp-dir\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.164665 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.164570    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-home\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.164733 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.164696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-tmp-dir\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.164797 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.164763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-model-cache\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.166437 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.166407    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-dshm\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.166783 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.166760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef085a68-d4f8-466f-b7dc-2d181289f25e-tls-certs\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.171996 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.171979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6dfs\" (UniqueName: \"kubernetes.io/projected/ef085a68-d4f8-466f-b7dc-2d181289f25e-kube-api-access-x6dfs\") pod \"distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") " pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.278090 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.278001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:26.550132 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:26.550107    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"]
Sep 15 12:48:26.551350 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:48:26.551325    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef085a68_d4f8_466f_b7dc_2d181289f25e.slice/crio-f13b34e51fabff6cb2b55b0f594d1d45ee42cf52983b5354acd8c80b969e8afa WatchSource:0}: Error finding container f13b34e51fabff6cb2b55b0f594d1d45ee42cf52983b5354acd8c80b969e8afa: Status 404 returned error can't find the container with id f13b34e51fabff6cb2b55b0f594d1d45ee42cf52983b5354acd8c80b969e8afa
Sep 15 12:48:27.041566 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:27.041536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q" event={"ID":"ef085a68-d4f8-466f-b7dc-2d181289f25e","Type":"ContainerStarted","Data":"f06f80399bb10eb5661d7eab450e1af71f1c7d2df52ed8f696d93590d1bcfe9f"}
Sep 15 12:48:27.041753 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:27.041577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q" event={"ID":"ef085a68-d4f8-466f-b7dc-2d181289f25e","Type":"ContainerStarted","Data":"f13b34e51fabff6cb2b55b0f594d1d45ee42cf52983b5354acd8c80b969e8afa"}
Sep 15 12:48:27.059305 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:27.059257    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q" podStartSLOduration=2.059238196 podStartE2EDuration="2.059238196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:48:27.058737367 +0000 UTC m=+1378.443448252" watchObservedRunningTime="2026-09-15 12:48:27.059238196 +0000 UTC m=+1378.443949080"
Sep 15 12:48:28.440475 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:28.440429    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"]
Sep 15 12:48:28.490255 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:28.490217    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:29.433947 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:29.433916    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"]
Sep 15 12:48:29.460665 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:29.460469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:33.617408 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:33.617367    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"]
Sep 15 12:48:33.617912 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:33.617625    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7" podUID="fa0cd320-b68d-49f8-bd29-fcfb91ba467f" containerName="maas-api" containerID="cri-o://d155f4496243ef5e679c3642b76f40350752046509c5401c0643b683d765a6b1" gracePeriod=30
Sep 15 12:48:33.692216 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:33.692181    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:33.804421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:33.804387    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:33.865490 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:33.865463    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:48:33.887424 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:33.887347    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:33.947090 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:33.947052    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:33.951774 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:33.951744    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-openshift-service-ca\") pod \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") "
Sep 15 12:48:33.951867 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:33.951796    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-metrics-certs\") pod \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") "
Sep 15 12:48:33.951929 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:33.951905    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5xdw\" (UniqueName: \"kubernetes.io/projected/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-kube-api-access-l5xdw\") pod \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") "
Sep 15 12:48:33.951929 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:33.951922    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-maas-api-tls\") pod \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\" (UID: \"fa0cd320-b68d-49f8-bd29-fcfb91ba467f\") "
Sep 15 12:48:33.952140 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:33.952116    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fa0cd320-b68d-49f8-bd29-fcfb91ba467f" (UID: "fa0cd320-b68d-49f8-bd29-fcfb91ba467f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:48:33.953973 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:33.953944    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fa0cd320-b68d-49f8-bd29-fcfb91ba467f" (UID: "fa0cd320-b68d-49f8-bd29-fcfb91ba467f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:48:33.954061 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:33.953969    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fa0cd320-b68d-49f8-bd29-fcfb91ba467f" (UID: "fa0cd320-b68d-49f8-bd29-fcfb91ba467f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:48:33.954061 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:33.954005    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-kube-api-access-l5xdw" (OuterVolumeSpecName: "kube-api-access-l5xdw") pod "fa0cd320-b68d-49f8-bd29-fcfb91ba467f" (UID: "fa0cd320-b68d-49f8-bd29-fcfb91ba467f"). InnerVolumeSpecName "kube-api-access-l5xdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:48:34.053114 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:34.053082    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-metrics-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:34.053114 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:34.053108    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5xdw\" (UniqueName: \"kubernetes.io/projected/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-kube-api-access-l5xdw\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:34.053114 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:34.053118    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-maas-api-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:34.053350 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:34.053128    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0cd320-b68d-49f8-bd29-fcfb91ba467f-openshift-service-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:34.083292 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:34.083253    2579 generic.go:356] "Generic (PLEG): container finished" podID="fa0cd320-b68d-49f8-bd29-fcfb91ba467f" containerID="d155f4496243ef5e679c3642b76f40350752046509c5401c0643b683d765a6b1" exitCode=0
Sep 15 12:48:34.083292 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:34.083282    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"
Sep 15 12:48:34.083472 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:34.083328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7" event={"ID":"fa0cd320-b68d-49f8-bd29-fcfb91ba467f","Type":"ContainerDied","Data":"d155f4496243ef5e679c3642b76f40350752046509c5401c0643b683d765a6b1"}
Sep 15 12:48:34.083472 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:34.083376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7" event={"ID":"fa0cd320-b68d-49f8-bd29-fcfb91ba467f","Type":"ContainerDied","Data":"06f8a0fb83518ed2d25d3fc0f2080590f781aa4c0e9292c1048b4f8f2b807227"}
Sep 15 12:48:34.083472 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:34.083402    2579 scope.go:117] "RemoveContainer" containerID="d155f4496243ef5e679c3642b76f40350752046509c5401c0643b683d765a6b1"
Sep 15 12:48:34.093358 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:34.093341    2579 scope.go:117] "RemoveContainer" containerID="d155f4496243ef5e679c3642b76f40350752046509c5401c0643b683d765a6b1"
Sep 15 12:48:34.093677 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:48:34.093649    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d155f4496243ef5e679c3642b76f40350752046509c5401c0643b683d765a6b1\": container with ID starting with d155f4496243ef5e679c3642b76f40350752046509c5401c0643b683d765a6b1 not found: ID does not exist" containerID="d155f4496243ef5e679c3642b76f40350752046509c5401c0643b683d765a6b1"
Sep 15 12:48:34.093776 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:34.093688    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d155f4496243ef5e679c3642b76f40350752046509c5401c0643b683d765a6b1"} err="failed to get container status \"d155f4496243ef5e679c3642b76f40350752046509c5401c0643b683d765a6b1\": rpc error: code = NotFound desc = could not find container \"d155f4496243ef5e679c3642b76f40350752046509c5401c0643b683d765a6b1\": container with ID starting with d155f4496243ef5e679c3642b76f40350752046509c5401c0643b683d765a6b1 not found: ID does not exist"
Sep 15 12:48:34.107093 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:34.107050    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"]
Sep 15 12:48:34.112050 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:34.112023    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-88615e8e-5dcdc994b6-z7dt7"]
Sep 15 12:48:34.394403 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:34.394369    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:35.105502 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:35.105472    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa0cd320-b68d-49f8-bd29-fcfb91ba467f" path="/var/lib/kubelet/pods/fa0cd320-b68d-49f8-bd29-fcfb91ba467f/volumes"
Sep 15 12:48:35.198144 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:35.198117    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:35.884826 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:35.884787    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:36.278280 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:36.278192    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:36.278280 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:36.278258    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:36.290601 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:36.290573    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:36.573489 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:36.573412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:37.072552 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:37.072517    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:37.113911 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:37.113881    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:48:37.250260 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:37.250228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:37.995947 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:37.995915    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:38.677207 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:38.677168    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:42.095555 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:42.095522    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:43.460196 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.460130    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd" podUID="ffb4c622-7abc-4313-bd77-c6ed9e0a38df" containerName="main" containerID="cri-o://bdc2602c7610366f4d70949de5db6f54f98afcc5755ac3c7b436b5a2a955c238" gracePeriod=45
Sep 15 12:48:43.706914 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.706888    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:48:43.847955 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.847922    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-tmp-dir\") pod \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") "
Sep 15 12:48:43.848097 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.847977    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-home\") pod \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") "
Sep 15 12:48:43.848097 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.848020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr8gq\" (UniqueName: \"kubernetes.io/projected/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-kube-api-access-xr8gq\") pod \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") "
Sep 15 12:48:43.848097 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.848042    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-tls-certs\") pod \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") "
Sep 15 12:48:43.848255 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.848098    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-model-cache\") pod \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") "
Sep 15 12:48:43.848255 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.848131    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-dshm\") pod \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\" (UID: \"ffb4c622-7abc-4313-bd77-c6ed9e0a38df\") "
Sep 15 12:48:43.848255 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.848193    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffb4c622-7abc-4313-bd77-c6ed9e0a38df" (UID: "ffb4c622-7abc-4313-bd77-c6ed9e0a38df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:48:43.848416 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.848248    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-home" (OuterVolumeSpecName: "home") pod "ffb4c622-7abc-4313-bd77-c6ed9e0a38df" (UID: "ffb4c622-7abc-4313-bd77-c6ed9e0a38df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:48:43.848416 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.848372    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffb4c622-7abc-4313-bd77-c6ed9e0a38df" (UID: "ffb4c622-7abc-4313-bd77-c6ed9e0a38df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:48:43.848513 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.848485    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:43.848513 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.848506    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:43.848581 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.848521    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:43.850289 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.850268    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-dshm" (OuterVolumeSpecName: "dshm") pod "ffb4c622-7abc-4313-bd77-c6ed9e0a38df" (UID: "ffb4c622-7abc-4313-bd77-c6ed9e0a38df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:48:43.850374 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.850332    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffb4c622-7abc-4313-bd77-c6ed9e0a38df" (UID: "ffb4c622-7abc-4313-bd77-c6ed9e0a38df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:48:43.850422 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.850395    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-kube-api-access-xr8gq" (OuterVolumeSpecName: "kube-api-access-xr8gq") pod "ffb4c622-7abc-4313-bd77-c6ed9e0a38df" (UID: "ffb4c622-7abc-4313-bd77-c6ed9e0a38df"). InnerVolumeSpecName "kube-api-access-xr8gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:48:43.949057 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.949025    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr8gq\" (UniqueName: \"kubernetes.io/projected/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-kube-api-access-xr8gq\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:43.949057 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.949052    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:43.949057 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:43.949062    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffb4c622-7abc-4313-bd77-c6ed9e0a38df-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:44.141149 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.141067    2579 generic.go:356] "Generic (PLEG): container finished" podID="ffb4c622-7abc-4313-bd77-c6ed9e0a38df" containerID="bdc2602c7610366f4d70949de5db6f54f98afcc5755ac3c7b436b5a2a955c238" exitCode=0
Sep 15 12:48:44.141149 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.141106    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"
Sep 15 12:48:44.141349 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.141148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd" event={"ID":"ffb4c622-7abc-4313-bd77-c6ed9e0a38df","Type":"ContainerDied","Data":"bdc2602c7610366f4d70949de5db6f54f98afcc5755ac3c7b436b5a2a955c238"}
Sep 15 12:48:44.141349 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.141192    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd" event={"ID":"ffb4c622-7abc-4313-bd77-c6ed9e0a38df","Type":"ContainerDied","Data":"8ae6691e505ac1c33337b5eaa16d1324bd28afe4a26c9c4666063435887e428a"}
Sep 15 12:48:44.141349 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.141208    2579 scope.go:117] "RemoveContainer" containerID="bdc2602c7610366f4d70949de5db6f54f98afcc5755ac3c7b436b5a2a955c238"
Sep 15 12:48:44.150630 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.150614    2579 scope.go:117] "RemoveContainer" containerID="bdc2602c7610366f4d70949de5db6f54f98afcc5755ac3c7b436b5a2a955c238"
Sep 15 12:48:44.150922 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:48:44.150902    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc2602c7610366f4d70949de5db6f54f98afcc5755ac3c7b436b5a2a955c238\": container with ID starting with bdc2602c7610366f4d70949de5db6f54f98afcc5755ac3c7b436b5a2a955c238 not found: ID does not exist" containerID="bdc2602c7610366f4d70949de5db6f54f98afcc5755ac3c7b436b5a2a955c238"
Sep 15 12:48:44.150974 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.150930    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc2602c7610366f4d70949de5db6f54f98afcc5755ac3c7b436b5a2a955c238"} err="failed to get container status \"bdc2602c7610366f4d70949de5db6f54f98afcc5755ac3c7b436b5a2a955c238\": rpc error: code = NotFound desc = could not find container \"bdc2602c7610366f4d70949de5db6f54f98afcc5755ac3c7b436b5a2a955c238\": container with ID starting with bdc2602c7610366f4d70949de5db6f54f98afcc5755ac3c7b436b5a2a955c238 not found: ID does not exist"
Sep 15 12:48:44.163254 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.163234    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"]
Sep 15 12:48:44.166890 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.166867    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-b-2381886b-kserve-ff98dccc5-hzkjd"]
Sep 15 12:48:44.453361 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.453247    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7" podUID="478306d3-9380-4336-b300-9d9f313eb0e1" containerName="main" containerID="cri-o://3a2a90f29fae653391b134f0f2f7d649157a5ab15414ad1c675af7633bb75100" gracePeriod=45
Sep 15 12:48:44.699874 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.699822    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:48:44.756017 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.755949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-dshm\") pod \"478306d3-9380-4336-b300-9d9f313eb0e1\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") "
Sep 15 12:48:44.756017 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.755990    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-home\") pod \"478306d3-9380-4336-b300-9d9f313eb0e1\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") "
Sep 15 12:48:44.756241 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.756065    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v9bk\" (UniqueName: \"kubernetes.io/projected/478306d3-9380-4336-b300-9d9f313eb0e1-kube-api-access-5v9bk\") pod \"478306d3-9380-4336-b300-9d9f313eb0e1\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") "
Sep 15 12:48:44.756241 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.756093    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-model-cache\") pod \"478306d3-9380-4336-b300-9d9f313eb0e1\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") "
Sep 15 12:48:44.756241 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.756133    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/478306d3-9380-4336-b300-9d9f313eb0e1-tls-certs\") pod \"478306d3-9380-4336-b300-9d9f313eb0e1\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") "
Sep 15 12:48:44.756241 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.756173    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-tmp-dir\") pod \"478306d3-9380-4336-b300-9d9f313eb0e1\" (UID: \"478306d3-9380-4336-b300-9d9f313eb0e1\") "
Sep 15 12:48:44.756449 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.756333    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-home" (OuterVolumeSpecName: "home") pod "478306d3-9380-4336-b300-9d9f313eb0e1" (UID: "478306d3-9380-4336-b300-9d9f313eb0e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:48:44.756449 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.756369    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "478306d3-9380-4336-b300-9d9f313eb0e1" (UID: "478306d3-9380-4336-b300-9d9f313eb0e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:48:44.756574 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.756550    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "478306d3-9380-4336-b300-9d9f313eb0e1" (UID: "478306d3-9380-4336-b300-9d9f313eb0e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:48:44.756641 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.756571    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:44.756641 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.756585    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:44.758238 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.758214    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-dshm" (OuterVolumeSpecName: "dshm") pod "478306d3-9380-4336-b300-9d9f313eb0e1" (UID: "478306d3-9380-4336-b300-9d9f313eb0e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:48:44.758356 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.758285    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/478306d3-9380-4336-b300-9d9f313eb0e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "478306d3-9380-4336-b300-9d9f313eb0e1" (UID: "478306d3-9380-4336-b300-9d9f313eb0e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:48:44.758356 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.758289    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478306d3-9380-4336-b300-9d9f313eb0e1-kube-api-access-5v9bk" (OuterVolumeSpecName: "kube-api-access-5v9bk") pod "478306d3-9380-4336-b300-9d9f313eb0e1" (UID: "478306d3-9380-4336-b300-9d9f313eb0e1"). InnerVolumeSpecName "kube-api-access-5v9bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:48:44.858009 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.857978    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:44.858009 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.858007    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v9bk\" (UniqueName: \"kubernetes.io/projected/478306d3-9380-4336-b300-9d9f313eb0e1-kube-api-access-5v9bk\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:44.858207 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.858019    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/478306d3-9380-4336-b300-9d9f313eb0e1-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:44.858207 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:44.858028    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/478306d3-9380-4336-b300-9d9f313eb0e1-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:48:45.105516 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:45.105489    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffb4c622-7abc-4313-bd77-c6ed9e0a38df" path="/var/lib/kubelet/pods/ffb4c622-7abc-4313-bd77-c6ed9e0a38df/volumes"
Sep 15 12:48:45.149565 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:45.149530    2579 generic.go:356] "Generic (PLEG): container finished" podID="478306d3-9380-4336-b300-9d9f313eb0e1" containerID="3a2a90f29fae653391b134f0f2f7d649157a5ab15414ad1c675af7633bb75100" exitCode=0
Sep 15 12:48:45.149700 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:45.149571    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"
Sep 15 12:48:45.149700 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:45.149584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7" event={"ID":"478306d3-9380-4336-b300-9d9f313eb0e1","Type":"ContainerDied","Data":"3a2a90f29fae653391b134f0f2f7d649157a5ab15414ad1c675af7633bb75100"}
Sep 15 12:48:45.149700 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:45.149620    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7" event={"ID":"478306d3-9380-4336-b300-9d9f313eb0e1","Type":"ContainerDied","Data":"f62465d6b9185ee492256d6548f17ce38795f5d72f641ddecc674fb721a78cf9"}
Sep 15 12:48:45.149700 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:45.149642    2579 scope.go:117] "RemoveContainer" containerID="3a2a90f29fae653391b134f0f2f7d649157a5ab15414ad1c675af7633bb75100"
Sep 15 12:48:45.158368 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:45.158348    2579 scope.go:117] "RemoveContainer" containerID="3a2a90f29fae653391b134f0f2f7d649157a5ab15414ad1c675af7633bb75100"
Sep 15 12:48:45.158636 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:48:45.158618    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2a90f29fae653391b134f0f2f7d649157a5ab15414ad1c675af7633bb75100\": container with ID starting with 3a2a90f29fae653391b134f0f2f7d649157a5ab15414ad1c675af7633bb75100 not found: ID does not exist" containerID="3a2a90f29fae653391b134f0f2f7d649157a5ab15414ad1c675af7633bb75100"
Sep 15 12:48:45.158684 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:45.158643    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2a90f29fae653391b134f0f2f7d649157a5ab15414ad1c675af7633bb75100"} err="failed to get container status \"3a2a90f29fae653391b134f0f2f7d649157a5ab15414ad1c675af7633bb75100\": rpc error: code = NotFound desc = could not find container \"3a2a90f29fae653391b134f0f2f7d649157a5ab15414ad1c675af7633bb75100\": container with ID starting with 3a2a90f29fae653391b134f0f2f7d649157a5ab15414ad1c675af7633bb75100 not found: ID does not exist"
Sep 15 12:48:45.165447 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:45.165426    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"]
Sep 15 12:48:45.169518 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:45.169499    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/e2e-conflict-a-2381886b-kserve-84b79c7b77-5stk7"]
Sep 15 12:48:46.547412 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.547374    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"]
Sep 15 12:48:46.547817 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.547749    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="478306d3-9380-4336-b300-9d9f313eb0e1" containerName="main"
Sep 15 12:48:46.547817 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.547761    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="478306d3-9380-4336-b300-9d9f313eb0e1" containerName="main"
Sep 15 12:48:46.547817 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.547782    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffb4c622-7abc-4313-bd77-c6ed9e0a38df" containerName="main"
Sep 15 12:48:46.547817 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.547787    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffb4c622-7abc-4313-bd77-c6ed9e0a38df" containerName="main"
Sep 15 12:48:46.547817 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.547795    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0cd320-b68d-49f8-bd29-fcfb91ba467f" containerName="maas-api"
Sep 15 12:48:46.547817 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.547801    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0cd320-b68d-49f8-bd29-fcfb91ba467f" containerName="maas-api"
Sep 15 12:48:46.548064 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.547877    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="478306d3-9380-4336-b300-9d9f313eb0e1" containerName="main"
Sep 15 12:48:46.548064 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.547884    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0cd320-b68d-49f8-bd29-fcfb91ba467f" containerName="maas-api"
Sep 15 12:48:46.548064 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.547890    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffb4c622-7abc-4313-bd77-c6ed9e0a38df" containerName="main"
Sep 15 12:48:46.552114 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.552096    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.556078 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.556042    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-29a9e9\"/\"default-dockercfg-gwmjv\""
Sep 15 12:48:46.556078 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.556063    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-29a9e9\"/\"auth-test-model-29a9e9-kserve-self-signed-certs\""
Sep 15 12:48:46.556274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.556112    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-29a9e9\"/\"openshift-service-ca.crt\""
Sep 15 12:48:46.556274 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.556071    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-29a9e9\"/\"kube-root-ca.crt\""
Sep 15 12:48:46.560985 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.560962    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"]
Sep 15 12:48:46.646970 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.646933    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:46.676425 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.676392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbvhb\" (UniqueName: \"kubernetes.io/projected/21f9fbfd-ebc8-4057-9a41-b842c8818968-kube-api-access-wbvhb\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.676586 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.676426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-model-cache\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.676586 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.676455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-home\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.676586 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.676510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-dshm\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.676715 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.676589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21f9fbfd-ebc8-4057-9a41-b842c8818968-tls-certs\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.676715 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.676615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-tmp-dir\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.778158 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.778122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-dshm\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.778158 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.778163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21f9fbfd-ebc8-4057-9a41-b842c8818968-tls-certs\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.778400 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.778181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-tmp-dir\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.778400 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.778264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbvhb\" (UniqueName: \"kubernetes.io/projected/21f9fbfd-ebc8-4057-9a41-b842c8818968-kube-api-access-wbvhb\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.778400 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.778290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-model-cache\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.778400 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.778317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-home\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.778676 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.778645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-tmp-dir\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.778676 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.778657    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-home\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.778866 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.778703    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-model-cache\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.780462 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.780436    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-dshm\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.780765 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.780749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21f9fbfd-ebc8-4057-9a41-b842c8818968-tls-certs\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.786541 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.786517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbvhb\" (UniqueName: \"kubernetes.io/projected/21f9fbfd-ebc8-4057-9a41-b842c8818968-kube-api-access-wbvhb\") pod \"auth-test-model-29a9e9-kserve-58685c778c-m9qkd\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:46.862866 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:46.862823    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:47.106065 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:47.106028    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="478306d3-9380-4336-b300-9d9f313eb0e1" path="/var/lib/kubelet/pods/478306d3-9380-4336-b300-9d9f313eb0e1/volumes"
Sep 15 12:48:47.135223 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:47.135148    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"]
Sep 15 12:48:47.138392 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:48:47.138367    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21f9fbfd_ebc8_4057_9a41_b842c8818968.slice/crio-34721edb24c87d1e547538fda521297f3f14bf7602c3e2a0e9bd60cb2ca6882e WatchSource:0}: Error finding container 34721edb24c87d1e547538fda521297f3f14bf7602c3e2a0e9bd60cb2ca6882e: Status 404 returned error can't find the container with id 34721edb24c87d1e547538fda521297f3f14bf7602c3e2a0e9bd60cb2ca6882e
Sep 15 12:48:47.165567 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:47.165540    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd" event={"ID":"21f9fbfd-ebc8-4057-9a41-b842c8818968","Type":"ContainerStarted","Data":"34721edb24c87d1e547538fda521297f3f14bf7602c3e2a0e9bd60cb2ca6882e"}
Sep 15 12:48:48.174271 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:48.174233    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd" event={"ID":"21f9fbfd-ebc8-4057-9a41-b842c8818968","Type":"ContainerStarted","Data":"5ef150bbc812ba52ef5f7a3e5995f9dec4ced108e3f548e70b24ddedfc053c45"}
Sep 15 12:48:48.190207 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:48.190150    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd" podStartSLOduration=2.190131752 podStartE2EDuration="2.190131752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:48:48.189464027 +0000 UTC m=+1399.574174910" watchObservedRunningTime="2026-09-15 12:48:48.190131752 +0000 UTC m=+1399.574842637"
Sep 15 12:48:56.863471 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:56.863435    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:56.864118 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:56.863514    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:56.875914 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:56.875893    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:56.954697 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:56.954659    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:48:57.244221 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:57.244135    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:48:57.800505 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:57.800456    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:48:59.772799 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:48:59.772764    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:49:18.469281 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:18.469239    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:49:18.992597 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:18.992560    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:49:19.556670 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:19.556631    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:49:24.164115 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.164034    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"]
Sep 15 12:49:24.167762 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.167744    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.170292 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.170266    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ee96ec\"/\"distinct-w4-ee96ec-kserve-self-signed-certs\""
Sep 15 12:49:24.176640 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.176619    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"]
Sep 15 12:49:24.184921 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.184896    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:49:24.241700 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.241670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-tmp-dir\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.241870 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.241705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-model-cache\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.241870 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.241727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-dshm\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.241967 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.241911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6tw7\" (UniqueName: \"kubernetes.io/projected/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-kube-api-access-b6tw7\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.242028 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.242003    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-tls-certs\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.242082 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.242055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-home\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.343055 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.343020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-tmp-dir\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.343055 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.343060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-model-cache\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.343302 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.343080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-dshm\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.343302 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.343116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tw7\" (UniqueName: \"kubernetes.io/projected/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-kube-api-access-b6tw7\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.343302 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.343167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-tls-certs\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.343302 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.343201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-home\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.343525 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.343484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-model-cache\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.343596 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.343578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-tmp-dir\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.343646 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.343609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-home\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.345431 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.345401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-dshm\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.345819 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.345798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-tls-certs\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.351002 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.350983    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tw7\" (UniqueName: \"kubernetes.io/projected/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-kube-api-access-b6tw7\") pod \"distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") " pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.479198 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.479104    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:24.666519 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.666456    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:49:24.752889 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:24.752861    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"]
Sep 15 12:49:24.754384 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:49:24.754354    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2baeee9d_c5c1_4ced_8df0_14c60444bfe4.slice/crio-a8a4c3664aaf9fad7536c478885fc4289e1da4d0b007ef88cd6c07bfbd560420 WatchSource:0}: Error finding container a8a4c3664aaf9fad7536c478885fc4289e1da4d0b007ef88cd6c07bfbd560420: Status 404 returned error can't find the container with id a8a4c3664aaf9fad7536c478885fc4289e1da4d0b007ef88cd6c07bfbd560420
Sep 15 12:49:25.399133 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:25.399094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp" event={"ID":"2baeee9d-c5c1-4ced-8df0-14c60444bfe4","Type":"ContainerStarted","Data":"6c2714d9d092fd3aebad4d154f9fb1827c03f5a5e53d4565f70c9c49e84d5cca"}
Sep 15 12:49:25.399511 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:25.399145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp" event={"ID":"2baeee9d-c5c1-4ced-8df0-14c60444bfe4","Type":"ContainerStarted","Data":"a8a4c3664aaf9fad7536c478885fc4289e1da4d0b007ef88cd6c07bfbd560420"}
Sep 15 12:49:25.416737 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:25.416692    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp" podStartSLOduration=1.416676488 podStartE2EDuration="1.416676488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:49:25.415347106 +0000 UTC m=+1436.800057990" watchObservedRunningTime="2026-09-15 12:49:25.416676488 +0000 UTC m=+1436.801387417"
Sep 15 12:49:29.143438 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:29.143377    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:49:34.480205 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:34.480158    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:34.480205 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:34.480222    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:34.492648 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:34.492624    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:35.465792 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:35.465765    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:49:45.882351 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:45.882309    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:49:45.983204 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:45.983161    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:49:48.728490 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:48.728451    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:49:48.794463 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:48.794428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:49:49.468684 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:49.468649    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:49:53.049663 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:53.049627    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:49:56.266270 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:56.266227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:49:56.327723 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:56.327695    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:49:57.056860 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:49:57.056813    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:50:03.074628 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:03.074575    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:05.912445 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:05.912412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:06.152110 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:06.152061    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:09.867713 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:09.867673    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:21.172627 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:21.172579    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:27.668721 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:27.668682    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:29.218246 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:29.218215    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:50:29.218772 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:29.218751    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:50:33.588098 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:33.588054    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:33.655594 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:33.655560    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:33.758195 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:33.758159    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:35.703697 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:35.703661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:36.370806 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:36.370773    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:37.078595 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:37.078560    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:50:37.324631 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:37.324595    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:37.421698 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:37.421662    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:38.280455 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:38.280423    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:39.077178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:39.077139    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:40.707391 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:40.707357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:40.810426 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:40.810391    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:41.273995 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.273965    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"]
Sep 15 12:50:41.277534 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.277510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:41.280488 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.280441    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-adc0cb-serving-cert\""
Sep 15 12:50:41.289445 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.289418    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"]
Sep 15 12:50:41.306728 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.306702    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:41.381529 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.381503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4123e2f2-87d1-41be-a62c-2d2854e52823-openshift-service-ca\") pod \"maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:41.381685 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.381571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbplf\" (UniqueName: \"kubernetes.io/projected/4123e2f2-87d1-41be-a62c-2d2854e52823-kube-api-access-tbplf\") pod \"maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:41.381685 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.381624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4123e2f2-87d1-41be-a62c-2d2854e52823-maas-api-tls\") pod \"maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:41.381685 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.381647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4123e2f2-87d1-41be-a62c-2d2854e52823-metrics-certs\") pod \"maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:41.407120 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.407087    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:41.483007 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.482972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbplf\" (UniqueName: \"kubernetes.io/projected/4123e2f2-87d1-41be-a62c-2d2854e52823-kube-api-access-tbplf\") pod \"maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:41.483180 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.483016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4123e2f2-87d1-41be-a62c-2d2854e52823-maas-api-tls\") pod \"maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:41.483180 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.483044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4123e2f2-87d1-41be-a62c-2d2854e52823-metrics-certs\") pod \"maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:41.483180 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:50:41.483132    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-serving-cert: secret "maas-api-e2e-api-a-adc0cb-serving-cert" not found
Sep 15 12:50:41.483180 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.483148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4123e2f2-87d1-41be-a62c-2d2854e52823-openshift-service-ca\") pod \"maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:41.483391 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:50:41.483197    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4123e2f2-87d1-41be-a62c-2d2854e52823-maas-api-tls podName:4123e2f2-87d1-41be-a62c-2d2854e52823 nodeName:}" failed. No retries permitted until 2026-09-15 12:50:41.983180827 +0000 UTC m=+1513.367891689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4123e2f2-87d1-41be-a62c-2d2854e52823-maas-api-tls") pod "maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk" (UID: "4123e2f2-87d1-41be-a62c-2d2854e52823") : secret "maas-api-e2e-api-a-adc0cb-serving-cert" not found
Sep 15 12:50:41.483843 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.483811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4123e2f2-87d1-41be-a62c-2d2854e52823-openshift-service-ca\") pod \"maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:41.485473 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.485456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4123e2f2-87d1-41be-a62c-2d2854e52823-metrics-certs\") pod \"maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:41.490846 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.490810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbplf\" (UniqueName: \"kubernetes.io/projected/4123e2f2-87d1-41be-a62c-2d2854e52823-kube-api-access-tbplf\") pod \"maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:41.746930 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.746882    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:41.878554 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.878520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:41.977078 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.977048    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:50:41.989286 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.989256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4123e2f2-87d1-41be-a62c-2d2854e52823-maas-api-tls\") pod \"maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:41.991614 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:41.991586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4123e2f2-87d1-41be-a62c-2d2854e52823-maas-api-tls\") pod \"maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:42.190320 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:42.190269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:42.462767 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:42.462728    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"]
Sep 15 12:50:42.464026 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:50:42.463998    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4123e2f2_87d1_41be_a62c_2d2854e52823.slice/crio-7d84f17c478ee0c7633d3319373a85e32337a9306c49fdfe2895b12a236e92d2 WatchSource:0}: Error finding container 7d84f17c478ee0c7633d3319373a85e32337a9306c49fdfe2895b12a236e92d2: Status 404 returned error can't find the container with id 7d84f17c478ee0c7633d3319373a85e32337a9306c49fdfe2895b12a236e92d2
Sep 15 12:50:42.832687 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:42.832598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk" event={"ID":"4123e2f2-87d1-41be-a62c-2d2854e52823","Type":"ContainerStarted","Data":"7d84f17c478ee0c7633d3319373a85e32337a9306c49fdfe2895b12a236e92d2"}
Sep 15 12:50:43.839491 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:43.839456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk" event={"ID":"4123e2f2-87d1-41be-a62c-2d2854e52823","Type":"ContainerStarted","Data":"c97c57862641d8f8b8e9c6ceb668f9d486d8300c5f38709a48432754186e9434"}
Sep 15 12:50:43.841825 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:43.839528    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:50:43.861362 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:43.861318    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk" podStartSLOduration=2.389925893 podStartE2EDuration="2.861302766s" totalImagesPullingTime="471.376873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:50:43.858898829 +0000 UTC m=+1515.243609713" watchObservedRunningTime="2026-09-15 12:50:43.861302766 +0000 UTC m=+1515.246013650"
Sep 15 12:50:49.850912 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:50:49.850877    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:51:03.402564 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:03.402527    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:04.174269 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:04.174234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:04.817815 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:04.817775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:05.471220 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:05.471184    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:06.281618 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:06.281574    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:06.925257 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:06.925222    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:07.025279 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:07.025236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:07.288491 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:07.288394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:07.606491 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:07.606457    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:07.707179 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:07.707145    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:08.597135 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:08.597098    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:09.368594 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:09.368559    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:10.081791 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:10.081746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:10.176459 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:10.176419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:10.990242 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:10.990204    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:11.778908 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:11.778869    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:12.552626 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:12.552587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:13.218688 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:13.218637    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:13.889070 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:13.889016    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:14.609605 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:14.609568    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:15.257264 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.257228    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"]
Sep 15 12:51:15.261790 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.261763    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:15.263953 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.263932    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-25e7f8-serving-cert\""
Sep 15 12:51:15.271641 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.271138    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"]
Sep 15 12:51:15.287282 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.287256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:15.305281 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.305249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s7r9\" (UniqueName: \"kubernetes.io/projected/bbc4ff18-d807-4c99-a70f-76e850dbd59a-kube-api-access-2s7r9\") pod \"maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:15.305405 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.305366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc4ff18-d807-4c99-a70f-76e850dbd59a-maas-api-tls\") pod \"maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:15.305405 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.305402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc4ff18-d807-4c99-a70f-76e850dbd59a-metrics-certs\") pod \"maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:15.305478 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.305421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc4ff18-d807-4c99-a70f-76e850dbd59a-openshift-service-ca\") pod \"maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:15.389439 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.389404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:15.406512 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.406487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc4ff18-d807-4c99-a70f-76e850dbd59a-maas-api-tls\") pod \"maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:15.406632 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.406518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc4ff18-d807-4c99-a70f-76e850dbd59a-metrics-certs\") pod \"maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:15.406632 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.406544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc4ff18-d807-4c99-a70f-76e850dbd59a-openshift-service-ca\") pod \"maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:15.406632 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.406587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s7r9\" (UniqueName: \"kubernetes.io/projected/bbc4ff18-d807-4c99-a70f-76e850dbd59a-kube-api-access-2s7r9\") pod \"maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:15.406737 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:51:15.406635    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-serving-cert: secret "maas-api-e2e-api-b-25e7f8-serving-cert" not found
Sep 15 12:51:15.407958 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:51:15.407194    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc4ff18-d807-4c99-a70f-76e850dbd59a-maas-api-tls podName:bbc4ff18-d807-4c99-a70f-76e850dbd59a nodeName:}" failed. No retries permitted until 2026-09-15 12:51:15.907164092 +0000 UTC m=+1547.291874963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bbc4ff18-d807-4c99-a70f-76e850dbd59a-maas-api-tls") pod "maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng" (UID: "bbc4ff18-d807-4c99-a70f-76e850dbd59a") : secret "maas-api-e2e-api-b-25e7f8-serving-cert" not found
Sep 15 12:51:15.407958 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.407853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc4ff18-d807-4c99-a70f-76e850dbd59a-openshift-service-ca\") pod \"maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:15.414298 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.410181    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc4ff18-d807-4c99-a70f-76e850dbd59a-metrics-certs\") pod \"maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:15.416970 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.416946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s7r9\" (UniqueName: \"kubernetes.io/projected/bbc4ff18-d807-4c99-a70f-76e850dbd59a-kube-api-access-2s7r9\") pod \"maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:15.509651 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.509549    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:15.834241 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.834146    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:15.912113 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.912078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc4ff18-d807-4c99-a70f-76e850dbd59a-maas-api-tls\") pod \"maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:15.914431 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.914411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc4ff18-d807-4c99-a70f-76e850dbd59a-maas-api-tls\") pod \"maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:15.934485 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:15.934454    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:16.175937 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:16.175905    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:16.447378 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:16.447351    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"]
Sep 15 12:51:16.449616 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:51:16.449588    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc4ff18_d807_4c99_a70f_76e850dbd59a.slice/crio-183485d04423291cc72c7d1094f8e968c1d63baa0361dd4f65f4f7c3e68f8189 WatchSource:0}: Error finding container 183485d04423291cc72c7d1094f8e968c1d63baa0361dd4f65f4f7c3e68f8189: Status 404 returned error can't find the container with id 183485d04423291cc72c7d1094f8e968c1d63baa0361dd4f65f4f7c3e68f8189
Sep 15 12:51:17.026589 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:17.026557    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng" event={"ID":"bbc4ff18-d807-4c99-a70f-76e850dbd59a","Type":"ContainerStarted","Data":"7d9b858fbee8fc676b7d9eab613c048feb2e91e3657d06a06c7a7977486fc4ff"}
Sep 15 12:51:17.027009 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:17.026607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng" event={"ID":"bbc4ff18-d807-4c99-a70f-76e850dbd59a","Type":"ContainerStarted","Data":"183485d04423291cc72c7d1094f8e968c1d63baa0361dd4f65f4f7c3e68f8189"}
Sep 15 12:51:17.027009 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:17.026642    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:17.042877 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:17.042802    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng" podStartSLOduration=1.674577339 podStartE2EDuration="2.042784451s" totalImagesPullingTime="368.207112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:51:17.042071469 +0000 UTC m=+1548.426782355" watchObservedRunningTime="2026-09-15 12:51:17.042784451 +0000 UTC m=+1548.427495335"
Sep 15 12:51:17.357104 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:17.357067    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:51:23.038268 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:23.038240    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:51:27.643136 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:27.643099    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:27.698074 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:27.698038    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:29.312075 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:29.312046    2579 scope.go:117] "RemoveContainer" containerID="da1b86020c88720e525aac7f0557a8c932dd58f7be548ef08fc92ae3b66b503b"
Sep 15 12:51:37.379798 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:37.379759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:38.042464 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:38.042423    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:38.677479 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:38.677442    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:39.397982 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:39.397921    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:39.994705 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:39.994668    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:40.094457 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:40.094424    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:40.657199 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:40.657163    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:40.991575 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:40.991484    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:41.678957 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:41.678920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:42.481846 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:42.481795    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:43.399479 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:43.399439    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:44.018719 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:44.018680    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:44.801549 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:44.801515    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:45.678104 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:45.678065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:46.647410 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:46.647371    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:47.104643 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:47.104593    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:48.003275 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:48.003234    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:48.722078 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:48.722046    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:49.486602 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:49.486564    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:53.086457 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:53.086425    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:58.650637 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:58.650605    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"]
Sep 15 12:51:58.651042 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:58.650813    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk" podUID="4123e2f2-87d1-41be-a62c-2d2854e52823" containerName="maas-api" containerID="cri-o://c97c57862641d8f8b8e9c6ceb668f9d486d8300c5f38709a48432754186e9434" gracePeriod=30
Sep 15 12:51:58.674755 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:58.674719    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:58.734332 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:58.734278    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:58.792812 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:58.792748    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:58.888484 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:58.888458    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:58.890613 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:58.890593    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:51:58.947595 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:58.947517    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:59.018635 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.018608    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4123e2f2-87d1-41be-a62c-2d2854e52823-openshift-service-ca\") pod \"4123e2f2-87d1-41be-a62c-2d2854e52823\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") "
Sep 15 12:51:59.018801 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.018789    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4123e2f2-87d1-41be-a62c-2d2854e52823-maas-api-tls\") pod \"4123e2f2-87d1-41be-a62c-2d2854e52823\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") "
Sep 15 12:51:59.018882 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.018813    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4123e2f2-87d1-41be-a62c-2d2854e52823-metrics-certs\") pod \"4123e2f2-87d1-41be-a62c-2d2854e52823\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") "
Sep 15 12:51:59.018882 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.018862    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbplf\" (UniqueName: \"kubernetes.io/projected/4123e2f2-87d1-41be-a62c-2d2854e52823-kube-api-access-tbplf\") pod \"4123e2f2-87d1-41be-a62c-2d2854e52823\" (UID: \"4123e2f2-87d1-41be-a62c-2d2854e52823\") "
Sep 15 12:51:59.019009 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.018988    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4123e2f2-87d1-41be-a62c-2d2854e52823-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4123e2f2-87d1-41be-a62c-2d2854e52823" (UID: "4123e2f2-87d1-41be-a62c-2d2854e52823"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:51:59.019172 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.019153    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4123e2f2-87d1-41be-a62c-2d2854e52823-openshift-service-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:51:59.020923 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.020900    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4123e2f2-87d1-41be-a62c-2d2854e52823-kube-api-access-tbplf" (OuterVolumeSpecName: "kube-api-access-tbplf") pod "4123e2f2-87d1-41be-a62c-2d2854e52823" (UID: "4123e2f2-87d1-41be-a62c-2d2854e52823"). InnerVolumeSpecName "kube-api-access-tbplf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:51:59.020923 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.020916    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4123e2f2-87d1-41be-a62c-2d2854e52823-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4123e2f2-87d1-41be-a62c-2d2854e52823" (UID: "4123e2f2-87d1-41be-a62c-2d2854e52823"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:51:59.021058 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.021014    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4123e2f2-87d1-41be-a62c-2d2854e52823-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4123e2f2-87d1-41be-a62c-2d2854e52823" (UID: "4123e2f2-87d1-41be-a62c-2d2854e52823"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:51:59.120019 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.119995    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4123e2f2-87d1-41be-a62c-2d2854e52823-maas-api-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:51:59.120019 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.120017    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4123e2f2-87d1-41be-a62c-2d2854e52823-metrics-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:51:59.120174 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.120027    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbplf\" (UniqueName: \"kubernetes.io/projected/4123e2f2-87d1-41be-a62c-2d2854e52823-kube-api-access-tbplf\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:51:59.270588 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.270501    2579 generic.go:356] "Generic (PLEG): container finished" podID="4123e2f2-87d1-41be-a62c-2d2854e52823" containerID="c97c57862641d8f8b8e9c6ceb668f9d486d8300c5f38709a48432754186e9434" exitCode=0
Sep 15 12:51:59.270588 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.270550    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"
Sep 15 12:51:59.270787 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.270586    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk" event={"ID":"4123e2f2-87d1-41be-a62c-2d2854e52823","Type":"ContainerDied","Data":"c97c57862641d8f8b8e9c6ceb668f9d486d8300c5f38709a48432754186e9434"}
Sep 15 12:51:59.270787 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.270640    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk" event={"ID":"4123e2f2-87d1-41be-a62c-2d2854e52823","Type":"ContainerDied","Data":"7d84f17c478ee0c7633d3319373a85e32337a9306c49fdfe2895b12a236e92d2"}
Sep 15 12:51:59.270787 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.270664    2579 scope.go:117] "RemoveContainer" containerID="c97c57862641d8f8b8e9c6ceb668f9d486d8300c5f38709a48432754186e9434"
Sep 15 12:51:59.280716 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.280698    2579 scope.go:117] "RemoveContainer" containerID="c97c57862641d8f8b8e9c6ceb668f9d486d8300c5f38709a48432754186e9434"
Sep 15 12:51:59.281052 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:51:59.281015    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97c57862641d8f8b8e9c6ceb668f9d486d8300c5f38709a48432754186e9434\": container with ID starting with c97c57862641d8f8b8e9c6ceb668f9d486d8300c5f38709a48432754186e9434 not found: ID does not exist" containerID="c97c57862641d8f8b8e9c6ceb668f9d486d8300c5f38709a48432754186e9434"
Sep 15 12:51:59.281197 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.281058    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97c57862641d8f8b8e9c6ceb668f9d486d8300c5f38709a48432754186e9434"} err="failed to get container status \"c97c57862641d8f8b8e9c6ceb668f9d486d8300c5f38709a48432754186e9434\": rpc error: code = NotFound desc = could not find container \"c97c57862641d8f8b8e9c6ceb668f9d486d8300c5f38709a48432754186e9434\": container with ID starting with c97c57862641d8f8b8e9c6ceb668f9d486d8300c5f38709a48432754186e9434 not found: ID does not exist"
Sep 15 12:51:59.288205 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.288185    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"]
Sep 15 12:51:59.290516 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.290497    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-adc0cb-896cdb5fb-h2qbk"]
Sep 15 12:51:59.748065 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.748022    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:51:59.848155 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:51:59.848116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:00.281810 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:00.281776    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:01.073975 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:01.073937    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:01.106414 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:01.106387    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4123e2f2-87d1-41be-a62c-2d2854e52823" path="/var/lib/kubelet/pods/4123e2f2-87d1-41be-a62c-2d2854e52823/volumes"
Sep 15 12:52:01.788773 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:01.788717    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:02.463773 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:02.463740    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:03.387074 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:03.387038    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:04.074312 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:04.074263    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:04.881585 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:04.881552    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:05.203170 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:05.203090    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:05.938699 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:05.938662    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:06.040470 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:06.040439    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:06.672779 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:06.672739    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:07.571784 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:07.571741    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:08.224374 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:08.224342    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:08.882739 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:08.882703    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:09.715494 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:09.715444    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:09.766464 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:09.766428    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:52:11.169902 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:11.169850    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:11.444019 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:11.443923    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:12.178594 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:12.178559    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:13.450144 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:13.450108    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:15.971396 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:15.971359    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:21.600249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.600172    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"]
Sep 15 12:52:21.600930 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.600411    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng" podUID="bbc4ff18-d807-4c99-a70f-76e850dbd59a" containerName="maas-api" containerID="cri-o://7d9b858fbee8fc676b7d9eab613c048feb2e91e3657d06a06c7a7977486fc4ff" gracePeriod=30
Sep 15 12:52:21.630499 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.630469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:21.697413 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.697374    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:21.766724 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.766472    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:21.832904 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.832866    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:21.853306 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.853243    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:52:21.891539 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.891505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:21.944569 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.944535    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc4ff18-d807-4c99-a70f-76e850dbd59a-maas-api-tls\") pod \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") "
Sep 15 12:52:21.944569 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.944574    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc4ff18-d807-4c99-a70f-76e850dbd59a-openshift-service-ca\") pod \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") "
Sep 15 12:52:21.944809 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.944599    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc4ff18-d807-4c99-a70f-76e850dbd59a-metrics-certs\") pod \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") "
Sep 15 12:52:21.944809 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.944708    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s7r9\" (UniqueName: \"kubernetes.io/projected/bbc4ff18-d807-4c99-a70f-76e850dbd59a-kube-api-access-2s7r9\") pod \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\" (UID: \"bbc4ff18-d807-4c99-a70f-76e850dbd59a\") "
Sep 15 12:52:21.945060 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.945034    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc4ff18-d807-4c99-a70f-76e850dbd59a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bbc4ff18-d807-4c99-a70f-76e850dbd59a" (UID: "bbc4ff18-d807-4c99-a70f-76e850dbd59a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:52:21.946878 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.946847    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc4ff18-d807-4c99-a70f-76e850dbd59a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bbc4ff18-d807-4c99-a70f-76e850dbd59a" (UID: "bbc4ff18-d807-4c99-a70f-76e850dbd59a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:52:21.946975 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.946914    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc4ff18-d807-4c99-a70f-76e850dbd59a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bbc4ff18-d807-4c99-a70f-76e850dbd59a" (UID: "bbc4ff18-d807-4c99-a70f-76e850dbd59a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:52:21.946975 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:21.946956    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc4ff18-d807-4c99-a70f-76e850dbd59a-kube-api-access-2s7r9" (OuterVolumeSpecName: "kube-api-access-2s7r9") pod "bbc4ff18-d807-4c99-a70f-76e850dbd59a" (UID: "bbc4ff18-d807-4c99-a70f-76e850dbd59a"). InnerVolumeSpecName "kube-api-access-2s7r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:52:22.045944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.045913    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc4ff18-d807-4c99-a70f-76e850dbd59a-maas-api-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:52:22.045944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.045938    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc4ff18-d807-4c99-a70f-76e850dbd59a-openshift-service-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:52:22.045944 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.045948    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc4ff18-d807-4c99-a70f-76e850dbd59a-metrics-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:52:22.046180 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.045957    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s7r9\" (UniqueName: \"kubernetes.io/projected/bbc4ff18-d807-4c99-a70f-76e850dbd59a-kube-api-access-2s7r9\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:52:22.181549 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.181452    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:22.273950 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.273919    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:22.413461 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.413429    2579 generic.go:356] "Generic (PLEG): container finished" podID="bbc4ff18-d807-4c99-a70f-76e850dbd59a" containerID="7d9b858fbee8fc676b7d9eab613c048feb2e91e3657d06a06c7a7977486fc4ff" exitCode=0
Sep 15 12:52:22.413631 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.413471    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"
Sep 15 12:52:22.413631 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.413508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng" event={"ID":"bbc4ff18-d807-4c99-a70f-76e850dbd59a","Type":"ContainerDied","Data":"7d9b858fbee8fc676b7d9eab613c048feb2e91e3657d06a06c7a7977486fc4ff"}
Sep 15 12:52:22.413631 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.413563    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng" event={"ID":"bbc4ff18-d807-4c99-a70f-76e850dbd59a","Type":"ContainerDied","Data":"183485d04423291cc72c7d1094f8e968c1d63baa0361dd4f65f4f7c3e68f8189"}
Sep 15 12:52:22.414902 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.413586    2579 scope.go:117] "RemoveContainer" containerID="7d9b858fbee8fc676b7d9eab613c048feb2e91e3657d06a06c7a7977486fc4ff"
Sep 15 12:52:22.424296 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.424280    2579 scope.go:117] "RemoveContainer" containerID="7d9b858fbee8fc676b7d9eab613c048feb2e91e3657d06a06c7a7977486fc4ff"
Sep 15 12:52:22.424515 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:52:22.424497    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d9b858fbee8fc676b7d9eab613c048feb2e91e3657d06a06c7a7977486fc4ff\": container with ID starting with 7d9b858fbee8fc676b7d9eab613c048feb2e91e3657d06a06c7a7977486fc4ff not found: ID does not exist" containerID="7d9b858fbee8fc676b7d9eab613c048feb2e91e3657d06a06c7a7977486fc4ff"
Sep 15 12:52:22.424584 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.424522    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9b858fbee8fc676b7d9eab613c048feb2e91e3657d06a06c7a7977486fc4ff"} err="failed to get container status \"7d9b858fbee8fc676b7d9eab613c048feb2e91e3657d06a06c7a7977486fc4ff\": rpc error: code = NotFound desc = could not find container \"7d9b858fbee8fc676b7d9eab613c048feb2e91e3657d06a06c7a7977486fc4ff\": container with ID starting with 7d9b858fbee8fc676b7d9eab613c048feb2e91e3657d06a06c7a7977486fc4ff not found: ID does not exist"
Sep 15 12:52:22.434823 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.434770    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"]
Sep 15 12:52:22.437150 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.437131    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-25e7f8-6fbfbcddcc-gvvng"]
Sep 15 12:52:22.949310 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:22.949269    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:23.106263 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:23.106223    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbc4ff18-d807-4c99-a70f-76e850dbd59a" path="/var/lib/kubelet/pods/bbc4ff18-d807-4c99-a70f-76e850dbd59a/volumes"
Sep 15 12:52:23.805409 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:23.805357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:24.597393 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:24.597355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:25.280643 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:25.280608    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:26.638736 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:26.638705    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:26.751277 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:26.751237    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:27.864469 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:27.864426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:27.921014 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:27.920978    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:28.123254 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:28.123169    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:28.763261 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:28.763225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:29.579355 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:29.579322    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:30.265722 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:30.265687    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:30.896509 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:30.896466    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:31.661694 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:31.661658    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:32.422211 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:32.422162    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:33.196437 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:33.196404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:33.851056 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:33.851019    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:39.122733 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.122690    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"]
Sep 15 12:52:39.123732 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.123700    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p" podUID="afd23e0f-7668-41d6-8f66-a8bffa2cc1f2" containerName="maas-api" containerID="cri-o://2de0bf4ae8a36e4d3eafd4ddaafaa253b5b16551f64dddde45b2acde6923c6d9" gracePeriod=30
Sep 15 12:52:39.150082 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.150046    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:39.227507 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.227477    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:39.289213 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.289181    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:39.370657 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.370633    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:52:39.390462 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.390377    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:39.423213 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.423187    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-metrics-certs\") pod \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") "
Sep 15 12:52:39.423344 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.423246    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7hzr\" (UniqueName: \"kubernetes.io/projected/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-kube-api-access-d7hzr\") pod \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") "
Sep 15 12:52:39.423344 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.423272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-openshift-service-ca\") pod \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") "
Sep 15 12:52:39.423344 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.423336    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-maas-api-tls\") pod \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\" (UID: \"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2\") "
Sep 15 12:52:39.423674 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.423649    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "afd23e0f-7668-41d6-8f66-a8bffa2cc1f2" (UID: "afd23e0f-7668-41d6-8f66-a8bffa2cc1f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:52:39.425438 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.425409    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "afd23e0f-7668-41d6-8f66-a8bffa2cc1f2" (UID: "afd23e0f-7668-41d6-8f66-a8bffa2cc1f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:52:39.425546 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.425433    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "afd23e0f-7668-41d6-8f66-a8bffa2cc1f2" (UID: "afd23e0f-7668-41d6-8f66-a8bffa2cc1f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:52:39.425546 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.425489    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-kube-api-access-d7hzr" (OuterVolumeSpecName: "kube-api-access-d7hzr") pod "afd23e0f-7668-41d6-8f66-a8bffa2cc1f2" (UID: "afd23e0f-7668-41d6-8f66-a8bffa2cc1f2"). InnerVolumeSpecName "kube-api-access-d7hzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:52:39.518597 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.518563    2579 generic.go:356] "Generic (PLEG): container finished" podID="afd23e0f-7668-41d6-8f66-a8bffa2cc1f2" containerID="2de0bf4ae8a36e4d3eafd4ddaafaa253b5b16551f64dddde45b2acde6923c6d9" exitCode=0
Sep 15 12:52:39.518789 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.518602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p" event={"ID":"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2","Type":"ContainerDied","Data":"2de0bf4ae8a36e4d3eafd4ddaafaa253b5b16551f64dddde45b2acde6923c6d9"}
Sep 15 12:52:39.518789 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.518609    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"
Sep 15 12:52:39.518789 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.518655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p" event={"ID":"afd23e0f-7668-41d6-8f66-a8bffa2cc1f2","Type":"ContainerDied","Data":"0779084d6e5a3c714e9d0291e4afec55e8a5367d719836c89f99a9edcdba2308"}
Sep 15 12:52:39.518789 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.518685    2579 scope.go:117] "RemoveContainer" containerID="2de0bf4ae8a36e4d3eafd4ddaafaa253b5b16551f64dddde45b2acde6923c6d9"
Sep 15 12:52:39.524312 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.524290    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-maas-api-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:52:39.524312 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.524311    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-metrics-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:52:39.524451 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.524321    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7hzr\" (UniqueName: \"kubernetes.io/projected/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-kube-api-access-d7hzr\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:52:39.524451 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.524330    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2-openshift-service-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:52:39.528702 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.528685    2579 scope.go:117] "RemoveContainer" containerID="2de0bf4ae8a36e4d3eafd4ddaafaa253b5b16551f64dddde45b2acde6923c6d9"
Sep 15 12:52:39.529003 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:52:39.528982    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de0bf4ae8a36e4d3eafd4ddaafaa253b5b16551f64dddde45b2acde6923c6d9\": container with ID starting with 2de0bf4ae8a36e4d3eafd4ddaafaa253b5b16551f64dddde45b2acde6923c6d9 not found: ID does not exist" containerID="2de0bf4ae8a36e4d3eafd4ddaafaa253b5b16551f64dddde45b2acde6923c6d9"
Sep 15 12:52:39.529062 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.529010    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de0bf4ae8a36e4d3eafd4ddaafaa253b5b16551f64dddde45b2acde6923c6d9"} err="failed to get container status \"2de0bf4ae8a36e4d3eafd4ddaafaa253b5b16551f64dddde45b2acde6923c6d9\": rpc error: code = NotFound desc = could not find container \"2de0bf4ae8a36e4d3eafd4ddaafaa253b5b16551f64dddde45b2acde6923c6d9\": container with ID starting with 2de0bf4ae8a36e4d3eafd4ddaafaa253b5b16551f64dddde45b2acde6923c6d9 not found: ID does not exist"
Sep 15 12:52:39.541268 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.541244    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"]
Sep 15 12:52:39.544818 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.544798    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-a1ba7b-8c447f9bb-ff47p"]
Sep 15 12:52:39.712699 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.712609    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:39.895303 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:39.895269    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:40.586563 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:40.586526    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:41.106632 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:41.106604    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afd23e0f-7668-41d6-8f66-a8bffa2cc1f2" path="/var/lib/kubelet/pods/afd23e0f-7668-41d6-8f66-a8bffa2cc1f2/volumes"
Sep 15 12:52:41.487756 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:41.487663    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:42.202075 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:42.202035    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:42.991736 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:42.991702    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:43.775407 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:43.775370    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:45.615062 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:45.615026    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:45.715335 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:45.715296    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:45.865733 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:45.865635    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:46.481134 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:46.481097    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:47.391243 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:47.391210    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:48.051379 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:48.051343    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:48.881537 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:48.881500    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:49.680512 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:49.680464    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:50.475217 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:50.475188    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:51.222287 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:51.222242    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:56.798432 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:56.798398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:56.874566 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:56.874532    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:56.937003 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:56.936974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:56.994258 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:56.994225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:57.062043 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:57.061963    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:57.350208 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:57.350176    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:57.490994 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:57.490960    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:58.268234 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:58.268201    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:59.073213 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:59.073174    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:52:59.757951 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:52:59.757912    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:00.392740 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:00.392706    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:02.752257 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:02.752221    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:53:03.203219 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:03.203186    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:03.271921 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:03.271888    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:03.497996 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:03.497908    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:20.849162 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:20.849129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:31.834297 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:31.834255    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:32.291436 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:32.291396    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:32.991309 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:32.991272    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:33.682364 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:33.682332    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:34.496317 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:34.496284    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:35.264601 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:35.264571    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:35.914960 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:35.914914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:36.817349 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:36.817315    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"]
Sep 15 12:53:36.821969 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:36.821944    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"]
Sep 15 12:53:36.840074 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:36.840046    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"]
Sep 15 12:53:36.845859 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:36.845821    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"]
Sep 15 12:53:36.891898 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:36.891867    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:44.153548 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:44.153513    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:53:47.469172 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:47.469130    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:47.540504 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:47.540473    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:47.602441 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:47.602404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:47.677208 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:47.677178    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:47.742376 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:47.742289    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:48.034220 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:48.034133    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:48.135319 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:48.135288    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:48.884011 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:48.883980    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:49.675347 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:49.675317    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:50.318933 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:50.318884    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:51.837629 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:51.837543    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q" podUID="ef085a68-d4f8-466f-b7dc-2d181289f25e" containerName="main" containerID="cri-o://f06f80399bb10eb5661d7eab450e1af71f1c7d2df52ed8f696d93590d1bcfe9f" gracePeriod=45
Sep 15 12:53:51.842069 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:51.841687    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q" podUID="44ccd6bb-d7db-461f-be67-a6f7312cb5b1" containerName="main" containerID="cri-o://1c9bd9af00fb502c2976eb6e2be14d1c0f6d592264ddcc691cb300048f88bca1" gracePeriod=45
Sep 15 12:53:51.864219 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:51.864191    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8" podUID="31ba55e7-ef19-4ade-935a-d4d0ed246c3a" containerName="main" containerID="cri-o://125a5bbf0363b35afe35d790eba26d58cd334f08a76343f28f62d9eeefcaac12" gracePeriod=45
Sep 15 12:53:51.864382 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:51.864353    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447" podUID="4d6d2376-d742-421c-bccf-0fcc689eb29b" containerName="main" containerID="cri-o://a6b84e8466c6443412f97671f1b7af3462a38f384a3b3a54b9c89e683e9e230b" gracePeriod=45
Sep 15 12:53:52.130386 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.130366    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:53:52.144943 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.144919    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:53:52.195015 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.194991    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:53:52.196987 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.196966    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:53:52.213670 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.213638    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-tmp-dir\") pod \"ef085a68-d4f8-466f-b7dc-2d181289f25e\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") "
Sep 15 12:53:52.213821 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.213683    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef085a68-d4f8-466f-b7dc-2d181289f25e-tls-certs\") pod \"ef085a68-d4f8-466f-b7dc-2d181289f25e\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") "
Sep 15 12:53:52.213821 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.213711    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-model-cache\") pod \"ef085a68-d4f8-466f-b7dc-2d181289f25e\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") "
Sep 15 12:53:52.213821 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.213751    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-dshm\") pod \"ef085a68-d4f8-466f-b7dc-2d181289f25e\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") "
Sep 15 12:53:52.213821 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.213793    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-dshm\") pod \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") "
Sep 15 12:53:52.214090 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.213825    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-model-cache\") pod \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") "
Sep 15 12:53:52.214090 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.213878    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnzdl\" (UniqueName: \"kubernetes.io/projected/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-kube-api-access-qnzdl\") pod \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") "
Sep 15 12:53:52.214090 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.213909    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-home\") pod \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") "
Sep 15 12:53:52.214090 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.213932    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6dfs\" (UniqueName: \"kubernetes.io/projected/ef085a68-d4f8-466f-b7dc-2d181289f25e-kube-api-access-x6dfs\") pod \"ef085a68-d4f8-466f-b7dc-2d181289f25e\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") "
Sep 15 12:53:52.214090 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.213938    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef085a68-d4f8-466f-b7dc-2d181289f25e" (UID: "ef085a68-d4f8-466f-b7dc-2d181289f25e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.214090 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.213985    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-home\") pod \"ef085a68-d4f8-466f-b7dc-2d181289f25e\" (UID: \"ef085a68-d4f8-466f-b7dc-2d181289f25e\") "
Sep 15 12:53:52.214090 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.214036    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-tmp-dir\") pod \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") "
Sep 15 12:53:52.214090 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.214081    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-tls-certs\") pod \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\" (UID: \"44ccd6bb-d7db-461f-be67-a6f7312cb5b1\") "
Sep 15 12:53:52.214581 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.214160    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "44ccd6bb-d7db-461f-be67-a6f7312cb5b1" (UID: "44ccd6bb-d7db-461f-be67-a6f7312cb5b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.214581 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.214474    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.214581 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.214491    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.214768 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.214740    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef085a68-d4f8-466f-b7dc-2d181289f25e" (UID: "ef085a68-d4f8-466f-b7dc-2d181289f25e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.215676 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.215642    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-home" (OuterVolumeSpecName: "home") pod "44ccd6bb-d7db-461f-be67-a6f7312cb5b1" (UID: "44ccd6bb-d7db-461f-be67-a6f7312cb5b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.216289 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.215860    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44ccd6bb-d7db-461f-be67-a6f7312cb5b1" (UID: "44ccd6bb-d7db-461f-be67-a6f7312cb5b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.217133 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.217109    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-home" (OuterVolumeSpecName: "home") pod "ef085a68-d4f8-466f-b7dc-2d181289f25e" (UID: "ef085a68-d4f8-466f-b7dc-2d181289f25e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.217899 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.217875    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-dshm" (OuterVolumeSpecName: "dshm") pod "ef085a68-d4f8-466f-b7dc-2d181289f25e" (UID: "ef085a68-d4f8-466f-b7dc-2d181289f25e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.218423 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.218338    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-kube-api-access-qnzdl" (OuterVolumeSpecName: "kube-api-access-qnzdl") pod "44ccd6bb-d7db-461f-be67-a6f7312cb5b1" (UID: "44ccd6bb-d7db-461f-be67-a6f7312cb5b1"). InnerVolumeSpecName "kube-api-access-qnzdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:53:52.218423 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.218367    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44ccd6bb-d7db-461f-be67-a6f7312cb5b1" (UID: "44ccd6bb-d7db-461f-be67-a6f7312cb5b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:53:52.218423 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.218402    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-dshm" (OuterVolumeSpecName: "dshm") pod "44ccd6bb-d7db-461f-be67-a6f7312cb5b1" (UID: "44ccd6bb-d7db-461f-be67-a6f7312cb5b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.220029 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.219992    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef085a68-d4f8-466f-b7dc-2d181289f25e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef085a68-d4f8-466f-b7dc-2d181289f25e" (UID: "ef085a68-d4f8-466f-b7dc-2d181289f25e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:53:52.220029 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.219997    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef085a68-d4f8-466f-b7dc-2d181289f25e-kube-api-access-x6dfs" (OuterVolumeSpecName: "kube-api-access-x6dfs") pod "ef085a68-d4f8-466f-b7dc-2d181289f25e" (UID: "ef085a68-d4f8-466f-b7dc-2d181289f25e"). InnerVolumeSpecName "kube-api-access-x6dfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:53:52.314983 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.314949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-tmp-dir\") pod \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") "
Sep 15 12:53:52.314983 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.314991    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-tls-certs\") pod \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") "
Sep 15 12:53:52.315243 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315039    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-tmp-dir\") pod \"4d6d2376-d742-421c-bccf-0fcc689eb29b\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") "
Sep 15 12:53:52.315243 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315056    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-dshm\") pod \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") "
Sep 15 12:53:52.315243 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315083    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-home\") pod \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") "
Sep 15 12:53:52.315243 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315106    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-home\") pod \"4d6d2376-d742-421c-bccf-0fcc689eb29b\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") "
Sep 15 12:53:52.315243 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315142    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d6d2376-d742-421c-bccf-0fcc689eb29b-tls-certs\") pod \"4d6d2376-d742-421c-bccf-0fcc689eb29b\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") "
Sep 15 12:53:52.315243 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315173    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhhkb\" (UniqueName: \"kubernetes.io/projected/4d6d2376-d742-421c-bccf-0fcc689eb29b-kube-api-access-zhhkb\") pod \"4d6d2376-d742-421c-bccf-0fcc689eb29b\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") "
Sep 15 12:53:52.315243 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315196    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-model-cache\") pod \"4d6d2376-d742-421c-bccf-0fcc689eb29b\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") "
Sep 15 12:53:52.315609 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315232    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31ba55e7-ef19-4ade-935a-d4d0ed246c3a" (UID: "31ba55e7-ef19-4ade-935a-d4d0ed246c3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.315609 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315253    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw8wc\" (UniqueName: \"kubernetes.io/projected/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-kube-api-access-sw8wc\") pod \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") "
Sep 15 12:53:52.315609 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315278    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-model-cache\") pod \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\" (UID: \"31ba55e7-ef19-4ade-935a-d4d0ed246c3a\") "
Sep 15 12:53:52.315609 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315301    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-dshm\") pod \"4d6d2376-d742-421c-bccf-0fcc689eb29b\" (UID: \"4d6d2376-d742-421c-bccf-0fcc689eb29b\") "
Sep 15 12:53:52.315609 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315312    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d6d2376-d742-421c-bccf-0fcc689eb29b" (UID: "4d6d2376-d742-421c-bccf-0fcc689eb29b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.315609 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315350    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-home" (OuterVolumeSpecName: "home") pod "31ba55e7-ef19-4ade-935a-d4d0ed246c3a" (UID: "31ba55e7-ef19-4ade-935a-d4d0ed246c3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.315609 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315532    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d6d2376-d742-421c-bccf-0fcc689eb29b" (UID: "4d6d2376-d742-421c-bccf-0fcc689eb29b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315624    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-home" (OuterVolumeSpecName: "home") pod "4d6d2376-d742-421c-bccf-0fcc689eb29b" (UID: "4d6d2376-d742-421c-bccf-0fcc689eb29b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315670    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef085a68-d4f8-466f-b7dc-2d181289f25e-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315691    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315703    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315715    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315729    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnzdl\" (UniqueName: \"kubernetes.io/projected/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-kube-api-access-qnzdl\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315743    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315756    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6dfs\" (UniqueName: \"kubernetes.io/projected/ef085a68-d4f8-466f-b7dc-2d181289f25e-kube-api-access-x6dfs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315770    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315782    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef085a68-d4f8-466f-b7dc-2d181289f25e-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315793    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315804    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315815    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-model-cache" (OuterVolumeSpecName: "model-cache") pod "31ba55e7-ef19-4ade-935a-d4d0ed246c3a" (UID: "31ba55e7-ef19-4ade-935a-d4d0ed246c3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315857    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ccd6bb-d7db-461f-be67-a6f7312cb5b1-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.316020 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.315872    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.317398 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.317373    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6d2376-d742-421c-bccf-0fcc689eb29b-kube-api-access-zhhkb" (OuterVolumeSpecName: "kube-api-access-zhhkb") pod "4d6d2376-d742-421c-bccf-0fcc689eb29b" (UID: "4d6d2376-d742-421c-bccf-0fcc689eb29b"). InnerVolumeSpecName "kube-api-access-zhhkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:53:52.317595 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.317573    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-dshm" (OuterVolumeSpecName: "dshm") pod "31ba55e7-ef19-4ade-935a-d4d0ed246c3a" (UID: "31ba55e7-ef19-4ade-935a-d4d0ed246c3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.317787 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.317756    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31ba55e7-ef19-4ade-935a-d4d0ed246c3a" (UID: "31ba55e7-ef19-4ade-935a-d4d0ed246c3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:53:52.317899 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.317767    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-dshm" (OuterVolumeSpecName: "dshm") pod "4d6d2376-d742-421c-bccf-0fcc689eb29b" (UID: "4d6d2376-d742-421c-bccf-0fcc689eb29b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:53:52.317899 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.317800    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-kube-api-access-sw8wc" (OuterVolumeSpecName: "kube-api-access-sw8wc") pod "31ba55e7-ef19-4ade-935a-d4d0ed246c3a" (UID: "31ba55e7-ef19-4ade-935a-d4d0ed246c3a"). InnerVolumeSpecName "kube-api-access-sw8wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:53:52.318018 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.318000    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6d2376-d742-421c-bccf-0fcc689eb29b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d6d2376-d742-421c-bccf-0fcc689eb29b" (UID: "4d6d2376-d742-421c-bccf-0fcc689eb29b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:53:52.416559 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.416472    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.416559 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.416501    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d6d2376-d742-421c-bccf-0fcc689eb29b-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.416559 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.416511    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhhkb\" (UniqueName: \"kubernetes.io/projected/4d6d2376-d742-421c-bccf-0fcc689eb29b-kube-api-access-zhhkb\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.416559 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.416523    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw8wc\" (UniqueName: \"kubernetes.io/projected/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-kube-api-access-sw8wc\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.416559 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.416532    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.416559 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.416541    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d6d2376-d742-421c-bccf-0fcc689eb29b-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.416559 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.416548    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.416559 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.416555    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ba55e7-ef19-4ade-935a-d4d0ed246c3a-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:53:52.932108 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.932081    2579 generic.go:356] "Generic (PLEG): container finished" podID="44ccd6bb-d7db-461f-be67-a6f7312cb5b1" containerID="1c9bd9af00fb502c2976eb6e2be14d1c0f6d592264ddcc691cb300048f88bca1" exitCode=0
Sep 15 12:53:52.932562 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.932145    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"
Sep 15 12:53:52.932562 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.932164    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q" event={"ID":"44ccd6bb-d7db-461f-be67-a6f7312cb5b1","Type":"ContainerDied","Data":"1c9bd9af00fb502c2976eb6e2be14d1c0f6d592264ddcc691cb300048f88bca1"}
Sep 15 12:53:52.932562 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.932204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q" event={"ID":"44ccd6bb-d7db-461f-be67-a6f7312cb5b1","Type":"ContainerDied","Data":"8de17bb73211c32a3e24ba1420b97c82e4a7cf1736acc7856390e2e316eadfd6"}
Sep 15 12:53:52.932562 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.932225    2579 scope.go:117] "RemoveContainer" containerID="1c9bd9af00fb502c2976eb6e2be14d1c0f6d592264ddcc691cb300048f88bca1"
Sep 15 12:53:52.933648 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.933626    2579 generic.go:356] "Generic (PLEG): container finished" podID="31ba55e7-ef19-4ade-935a-d4d0ed246c3a" containerID="125a5bbf0363b35afe35d790eba26d58cd334f08a76343f28f62d9eeefcaac12" exitCode=0
Sep 15 12:53:52.933724 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.933681    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"
Sep 15 12:53:52.933724 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.933707    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8" event={"ID":"31ba55e7-ef19-4ade-935a-d4d0ed246c3a","Type":"ContainerDied","Data":"125a5bbf0363b35afe35d790eba26d58cd334f08a76343f28f62d9eeefcaac12"}
Sep 15 12:53:52.933862 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.933744    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8" event={"ID":"31ba55e7-ef19-4ade-935a-d4d0ed246c3a","Type":"ContainerDied","Data":"306a5b9674b99554ad5d40692c246713501be3c295846c2b1d51d09d244df7ab"}
Sep 15 12:53:52.935421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.935308    2579 generic.go:356] "Generic (PLEG): container finished" podID="ef085a68-d4f8-466f-b7dc-2d181289f25e" containerID="f06f80399bb10eb5661d7eab450e1af71f1c7d2df52ed8f696d93590d1bcfe9f" exitCode=0
Sep 15 12:53:52.935421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.935374    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"
Sep 15 12:53:52.935421 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.935394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q" event={"ID":"ef085a68-d4f8-466f-b7dc-2d181289f25e","Type":"ContainerDied","Data":"f06f80399bb10eb5661d7eab450e1af71f1c7d2df52ed8f696d93590d1bcfe9f"}
Sep 15 12:53:52.935646 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.935430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q" event={"ID":"ef085a68-d4f8-466f-b7dc-2d181289f25e","Type":"ContainerDied","Data":"f13b34e51fabff6cb2b55b0f594d1d45ee42cf52983b5354acd8c80b969e8afa"}
Sep 15 12:53:52.937303 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.937277    2579 generic.go:356] "Generic (PLEG): container finished" podID="4d6d2376-d742-421c-bccf-0fcc689eb29b" containerID="a6b84e8466c6443412f97671f1b7af3462a38f384a3b3a54b9c89e683e9e230b" exitCode=0
Sep 15 12:53:52.937408 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.937307    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447" event={"ID":"4d6d2376-d742-421c-bccf-0fcc689eb29b","Type":"ContainerDied","Data":"a6b84e8466c6443412f97671f1b7af3462a38f384a3b3a54b9c89e683e9e230b"}
Sep 15 12:53:52.937408 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.937316    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"
Sep 15 12:53:52.937408 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.937330    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447" event={"ID":"4d6d2376-d742-421c-bccf-0fcc689eb29b","Type":"ContainerDied","Data":"edaa2b387be5a5f50aeda5fca6492fe6df2ed41e4c8bfcb392749ce712e0861f"}
Sep 15 12:53:52.944613 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.944579    2579 scope.go:117] "RemoveContainer" containerID="1c9bd9af00fb502c2976eb6e2be14d1c0f6d592264ddcc691cb300048f88bca1"
Sep 15 12:53:52.945255 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:53:52.944901    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9bd9af00fb502c2976eb6e2be14d1c0f6d592264ddcc691cb300048f88bca1\": container with ID starting with 1c9bd9af00fb502c2976eb6e2be14d1c0f6d592264ddcc691cb300048f88bca1 not found: ID does not exist" containerID="1c9bd9af00fb502c2976eb6e2be14d1c0f6d592264ddcc691cb300048f88bca1"
Sep 15 12:53:52.945255 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.944937    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9bd9af00fb502c2976eb6e2be14d1c0f6d592264ddcc691cb300048f88bca1"} err="failed to get container status \"1c9bd9af00fb502c2976eb6e2be14d1c0f6d592264ddcc691cb300048f88bca1\": rpc error: code = NotFound desc = could not find container \"1c9bd9af00fb502c2976eb6e2be14d1c0f6d592264ddcc691cb300048f88bca1\": container with ID starting with 1c9bd9af00fb502c2976eb6e2be14d1c0f6d592264ddcc691cb300048f88bca1 not found: ID does not exist"
Sep 15 12:53:52.945255 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.944958    2579 scope.go:117] "RemoveContainer" containerID="125a5bbf0363b35afe35d790eba26d58cd334f08a76343f28f62d9eeefcaac12"
Sep 15 12:53:52.955514 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.955496    2579 scope.go:117] "RemoveContainer" containerID="125a5bbf0363b35afe35d790eba26d58cd334f08a76343f28f62d9eeefcaac12"
Sep 15 12:53:52.955768 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:53:52.955748    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"125a5bbf0363b35afe35d790eba26d58cd334f08a76343f28f62d9eeefcaac12\": container with ID starting with 125a5bbf0363b35afe35d790eba26d58cd334f08a76343f28f62d9eeefcaac12 not found: ID does not exist" containerID="125a5bbf0363b35afe35d790eba26d58cd334f08a76343f28f62d9eeefcaac12"
Sep 15 12:53:52.955869 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.955774    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125a5bbf0363b35afe35d790eba26d58cd334f08a76343f28f62d9eeefcaac12"} err="failed to get container status \"125a5bbf0363b35afe35d790eba26d58cd334f08a76343f28f62d9eeefcaac12\": rpc error: code = NotFound desc = could not find container \"125a5bbf0363b35afe35d790eba26d58cd334f08a76343f28f62d9eeefcaac12\": container with ID starting with 125a5bbf0363b35afe35d790eba26d58cd334f08a76343f28f62d9eeefcaac12 not found: ID does not exist"
Sep 15 12:53:52.955869 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.955791    2579 scope.go:117] "RemoveContainer" containerID="f06f80399bb10eb5661d7eab450e1af71f1c7d2df52ed8f696d93590d1bcfe9f"
Sep 15 12:53:52.958871 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.958847    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"]
Sep 15 12:53:52.963366 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.963345    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/premium-w0-11f865-kserve-dd78c4cc9-gvhr8"]
Sep 15 12:53:52.965762 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.965741    2579 scope.go:117] "RemoveContainer" containerID="f06f80399bb10eb5661d7eab450e1af71f1c7d2df52ed8f696d93590d1bcfe9f"
Sep 15 12:53:52.966025 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:53:52.966008    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06f80399bb10eb5661d7eab450e1af71f1c7d2df52ed8f696d93590d1bcfe9f\": container with ID starting with f06f80399bb10eb5661d7eab450e1af71f1c7d2df52ed8f696d93590d1bcfe9f not found: ID does not exist" containerID="f06f80399bb10eb5661d7eab450e1af71f1c7d2df52ed8f696d93590d1bcfe9f"
Sep 15 12:53:52.966090 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.966031    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06f80399bb10eb5661d7eab450e1af71f1c7d2df52ed8f696d93590d1bcfe9f"} err="failed to get container status \"f06f80399bb10eb5661d7eab450e1af71f1c7d2df52ed8f696d93590d1bcfe9f\": rpc error: code = NotFound desc = could not find container \"f06f80399bb10eb5661d7eab450e1af71f1c7d2df52ed8f696d93590d1bcfe9f\": container with ID starting with f06f80399bb10eb5661d7eab450e1af71f1c7d2df52ed8f696d93590d1bcfe9f not found: ID does not exist"
Sep 15 12:53:52.966090 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.966047    2579 scope.go:117] "RemoveContainer" containerID="a6b84e8466c6443412f97671f1b7af3462a38f384a3b3a54b9c89e683e9e230b"
Sep 15 12:53:52.971910 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.971888    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"]
Sep 15 12:53:52.974605 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.974582    2579 scope.go:117] "RemoveContainer" containerID="a6b84e8466c6443412f97671f1b7af3462a38f384a3b3a54b9c89e683e9e230b"
Sep 15 12:53:52.974956 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:53:52.974936    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b84e8466c6443412f97671f1b7af3462a38f384a3b3a54b9c89e683e9e230b\": container with ID starting with a6b84e8466c6443412f97671f1b7af3462a38f384a3b3a54b9c89e683e9e230b not found: ID does not exist" containerID="a6b84e8466c6443412f97671f1b7af3462a38f384a3b3a54b9c89e683e9e230b"
Sep 15 12:53:52.975047 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.974964    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b84e8466c6443412f97671f1b7af3462a38f384a3b3a54b9c89e683e9e230b"} err="failed to get container status \"a6b84e8466c6443412f97671f1b7af3462a38f384a3b3a54b9c89e683e9e230b\": rpc error: code = NotFound desc = could not find container \"a6b84e8466c6443412f97671f1b7af3462a38f384a3b3a54b9c89e683e9e230b\": container with ID starting with a6b84e8466c6443412f97671f1b7af3462a38f384a3b3a54b9c89e683e9e230b not found: ID does not exist"
Sep 15 12:53:52.975278 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.975260    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/distinct-w0-11f865-kserve-586d469ddb-pbd6q"]
Sep 15 12:53:52.983880 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.983862    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"]
Sep 15 12:53:52.991789 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:52.991770    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/simulator-w0-11f865-kserve-5b5f9c4966-2c447"]
Sep 15 12:53:53.001298 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:53.001277    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"]
Sep 15 12:53:53.005129 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:53.005109    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-11f865/distinct-2-w0-11f865-kserve-7cb7565bb5-g6g4q"]
Sep 15 12:53:53.106146 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:53.106073    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ba55e7-ef19-4ade-935a-d4d0ed246c3a" path="/var/lib/kubelet/pods/31ba55e7-ef19-4ade-935a-d4d0ed246c3a/volumes"
Sep 15 12:53:53.106485 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:53.106473    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44ccd6bb-d7db-461f-be67-a6f7312cb5b1" path="/var/lib/kubelet/pods/44ccd6bb-d7db-461f-be67-a6f7312cb5b1/volumes"
Sep 15 12:53:53.106817 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:53.106806    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d6d2376-d742-421c-bccf-0fcc689eb29b" path="/var/lib/kubelet/pods/4d6d2376-d742-421c-bccf-0fcc689eb29b/volumes"
Sep 15 12:53:53.107459 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:53.107444    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef085a68-d4f8-466f-b7dc-2d181289f25e" path="/var/lib/kubelet/pods/ef085a68-d4f8-466f-b7dc-2d181289f25e/volumes"
Sep 15 12:53:53.623889 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:53.623824    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:53.682500 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:53.682469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:53:53.880582 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:53:53.880498    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:00.469346 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:00.469308    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:02.874917 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:02.874882    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:03.996686 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:03.996652    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:04.658573 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:04.658540    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:05.281898 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:05.281862    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:06.084161 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:06.084128    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:07.859171 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:07.859135    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"]
Sep 15 12:54:07.869840 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:07.869802    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"]
Sep 15 12:54:07.879405 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:07.879372    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"]
Sep 15 12:54:07.941573 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:07.941535    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:12.449304 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:12.449268    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:54:13.827369 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:13.827327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:13.926777 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:13.926731    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:14.535225 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:14.535188    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:18.540244 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:18.540213    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ee96ec-29824605-lt7p8"]
Sep 15 12:54:18.540705 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:18.540508    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:18.543607 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:18.543585    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ee96ec-29824605-lt7p8"]
Sep 15 12:54:18.613401 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:18.613370    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:19.105743 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:19.105711    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cedf67b2-3bce-4f30-b0d5-9a7a195ca694" path="/var/lib/kubelet/pods/cedf67b2-3bce-4f30-b0d5-9a7a195ca694/volumes"
Sep 15 12:54:21.079348 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:21.079310    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:21.252572 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:21.252539    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:22.878875 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:22.878818    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp" podUID="2baeee9d-c5c1-4ced-8df0-14c60444bfe4" containerName="main" containerID="cri-o://6c2714d9d092fd3aebad4d154f9fb1827c03f5a5e53d4565f70c9c49e84d5cca" gracePeriod=45
Sep 15 12:54:22.890543 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:22.890509    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch" podUID="9bd21233-aa3a-48d6-8e3e-5ecc3fe79170" containerName="main" containerID="cri-o://3d98ae5fd23c5e19bfb41e47aa58c099a7bd13a793b405972c3ba704f5a6a283" gracePeriod=45
Sep 15 12:54:22.899006 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:22.898978    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86" podUID="9a45c3a4-8f63-4f88-ae6f-80d2ee45d492" containerName="main" containerID="cri-o://56395ca921ddc6b3ebf7bd1f4d38236ae9fd82a53a897ea53e9310f181037b31" gracePeriod=45
Sep 15 12:54:23.111767 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.111692    2579 generic.go:356] "Generic (PLEG): container finished" podID="9bd21233-aa3a-48d6-8e3e-5ecc3fe79170" containerID="3d98ae5fd23c5e19bfb41e47aa58c099a7bd13a793b405972c3ba704f5a6a283" exitCode=0
Sep 15 12:54:23.112165 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.112094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch" event={"ID":"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170","Type":"ContainerDied","Data":"3d98ae5fd23c5e19bfb41e47aa58c099a7bd13a793b405972c3ba704f5a6a283"}
Sep 15 12:54:23.114596 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.114514    2579 generic.go:356] "Generic (PLEG): container finished" podID="2baeee9d-c5c1-4ced-8df0-14c60444bfe4" containerID="6c2714d9d092fd3aebad4d154f9fb1827c03f5a5e53d4565f70c9c49e84d5cca" exitCode=0
Sep 15 12:54:23.114596 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.114554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp" event={"ID":"2baeee9d-c5c1-4ced-8df0-14c60444bfe4","Type":"ContainerDied","Data":"6c2714d9d092fd3aebad4d154f9fb1827c03f5a5e53d4565f70c9c49e84d5cca"}
Sep 15 12:54:23.116576 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.116552    2579 generic.go:356] "Generic (PLEG): container finished" podID="9a45c3a4-8f63-4f88-ae6f-80d2ee45d492" containerID="56395ca921ddc6b3ebf7bd1f4d38236ae9fd82a53a897ea53e9310f181037b31" exitCode=0
Sep 15 12:54:23.116663 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.116604    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86" event={"ID":"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492","Type":"ContainerDied","Data":"56395ca921ddc6b3ebf7bd1f4d38236ae9fd82a53a897ea53e9310f181037b31"}
Sep 15 12:54:23.171610 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.171589    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:54:23.174637 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.174609    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:54:23.178127 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.178108    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202147    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bst2d\" (UniqueName: \"kubernetes.io/projected/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-kube-api-access-bst2d\") pod \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202284    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-model-cache\") pod \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202326    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-tls-certs\") pod \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202359    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-home\") pod \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202395    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-tmp-dir\") pod \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202431    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-tmp-dir\") pod \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202450    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-model-cache\") pod \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202484    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npktd\" (UniqueName: \"kubernetes.io/projected/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-kube-api-access-npktd\") pod \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202507    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-dshm\") pod \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202539    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-tls-certs\") pod \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202557    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-dshm\") pod \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202581    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-model-cache\") pod \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202600    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-home\") pod \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202641    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6tw7\" (UniqueName: \"kubernetes.io/projected/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-kube-api-access-b6tw7\") pod \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\" (UID: \"2baeee9d-c5c1-4ced-8df0-14c60444bfe4\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202675    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-tls-certs\") pod \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202723    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-home\") pod \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\" (UID: \"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202749    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-dshm\") pod \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202770    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-tmp-dir\") pod \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\" (UID: \"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170\") "
Sep 15 12:54:23.202178 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.202821    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a45c3a4-8f63-4f88-ae6f-80d2ee45d492" (UID: "9a45c3a4-8f63-4f88-ae6f-80d2ee45d492"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:23.206249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.203015    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-model-cache" (OuterVolumeSpecName: "model-cache") pod "2baeee9d-c5c1-4ced-8df0-14c60444bfe4" (UID: "2baeee9d-c5c1-4ced-8df0-14c60444bfe4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:23.206249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.203058    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9a45c3a4-8f63-4f88-ae6f-80d2ee45d492" (UID: "9a45c3a4-8f63-4f88-ae6f-80d2ee45d492"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:23.206249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.203118    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-model-cache" (OuterVolumeSpecName: "model-cache") pod "9bd21233-aa3a-48d6-8e3e-5ecc3fe79170" (UID: "9bd21233-aa3a-48d6-8e3e-5ecc3fe79170"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:23.206249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.203238    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.206249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.203256    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.206249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.203272    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.206249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.203287    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.206249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.203276    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9bd21233-aa3a-48d6-8e3e-5ecc3fe79170" (UID: "9bd21233-aa3a-48d6-8e3e-5ecc3fe79170"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:23.206249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.204448    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-home" (OuterVolumeSpecName: "home") pod "2baeee9d-c5c1-4ced-8df0-14c60444bfe4" (UID: "2baeee9d-c5c1-4ced-8df0-14c60444bfe4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:23.206249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.204643    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-home" (OuterVolumeSpecName: "home") pod "9bd21233-aa3a-48d6-8e3e-5ecc3fe79170" (UID: "9bd21233-aa3a-48d6-8e3e-5ecc3fe79170"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:23.206249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.204934    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-home" (OuterVolumeSpecName: "home") pod "9a45c3a4-8f63-4f88-ae6f-80d2ee45d492" (UID: "9a45c3a4-8f63-4f88-ae6f-80d2ee45d492"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:23.206249 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.205733    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a45c3a4-8f63-4f88-ae6f-80d2ee45d492" (UID: "9a45c3a4-8f63-4f88-ae6f-80d2ee45d492"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:54:23.207229 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.206923    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2baeee9d-c5c1-4ced-8df0-14c60444bfe4" (UID: "2baeee9d-c5c1-4ced-8df0-14c60444bfe4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:23.208212 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.208186    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-dshm" (OuterVolumeSpecName: "dshm") pod "9bd21233-aa3a-48d6-8e3e-5ecc3fe79170" (UID: "9bd21233-aa3a-48d6-8e3e-5ecc3fe79170"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:23.208595 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.208567    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-kube-api-access-b6tw7" (OuterVolumeSpecName: "kube-api-access-b6tw7") pod "2baeee9d-c5c1-4ced-8df0-14c60444bfe4" (UID: "2baeee9d-c5c1-4ced-8df0-14c60444bfe4"). InnerVolumeSpecName "kube-api-access-b6tw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:54:23.209912 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.209889    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-kube-api-access-bst2d" (OuterVolumeSpecName: "kube-api-access-bst2d") pod "9bd21233-aa3a-48d6-8e3e-5ecc3fe79170" (UID: "9bd21233-aa3a-48d6-8e3e-5ecc3fe79170"). InnerVolumeSpecName "kube-api-access-bst2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:54:23.210632 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.210608    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-dshm" (OuterVolumeSpecName: "dshm") pod "9a45c3a4-8f63-4f88-ae6f-80d2ee45d492" (UID: "9a45c3a4-8f63-4f88-ae6f-80d2ee45d492"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:23.210734 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.210680    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9bd21233-aa3a-48d6-8e3e-5ecc3fe79170" (UID: "9bd21233-aa3a-48d6-8e3e-5ecc3fe79170"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:54:23.212008 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.211984    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-dshm" (OuterVolumeSpecName: "dshm") pod "2baeee9d-c5c1-4ced-8df0-14c60444bfe4" (UID: "2baeee9d-c5c1-4ced-8df0-14c60444bfe4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:23.212232 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.212201    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-kube-api-access-npktd" (OuterVolumeSpecName: "kube-api-access-npktd") pod "9a45c3a4-8f63-4f88-ae6f-80d2ee45d492" (UID: "9a45c3a4-8f63-4f88-ae6f-80d2ee45d492"). InnerVolumeSpecName "kube-api-access-npktd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:54:23.212324 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.212227    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2baeee9d-c5c1-4ced-8df0-14c60444bfe4" (UID: "2baeee9d-c5c1-4ced-8df0-14c60444bfe4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:54:23.304205 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.304169    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bst2d\" (UniqueName: \"kubernetes.io/projected/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-kube-api-access-bst2d\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.304205 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.304199    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.304205 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.304209    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.304433 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.304217    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.304433 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.304225    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npktd\" (UniqueName: \"kubernetes.io/projected/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-kube-api-access-npktd\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.304433 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.304232    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.304433 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.304240    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.304433 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.304248    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.304433 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.304255    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.304433 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.304263    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6tw7\" (UniqueName: \"kubernetes.io/projected/2baeee9d-c5c1-4ced-8df0-14c60444bfe4-kube-api-access-b6tw7\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.304433 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.304271    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.304433 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.304279    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.304433 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.304285    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:23.304433 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:23.304292    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:24.123181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.123148    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"
Sep 15 12:54:24.123181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.123168    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch" event={"ID":"9bd21233-aa3a-48d6-8e3e-5ecc3fe79170","Type":"ContainerDied","Data":"d73177bf26eca159e76841c821e1b062fc6529cdd1f9a898d9a4cef239ca9b79"}
Sep 15 12:54:24.123738 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.123216    2579 scope.go:117] "RemoveContainer" containerID="3d98ae5fd23c5e19bfb41e47aa58c099a7bd13a793b405972c3ba704f5a6a283"
Sep 15 12:54:24.124679 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.124660    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"
Sep 15 12:54:24.124765 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.124704    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp" event={"ID":"2baeee9d-c5c1-4ced-8df0-14c60444bfe4","Type":"ContainerDied","Data":"a8a4c3664aaf9fad7536c478885fc4289e1da4d0b007ef88cd6c07bfbd560420"}
Sep 15 12:54:24.126491 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.126464    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86" event={"ID":"9a45c3a4-8f63-4f88-ae6f-80d2ee45d492","Type":"ContainerDied","Data":"be317ebc3313584514669846ee1984f449734e0c0376c29c1f2ae7fe962244a1"}
Sep 15 12:54:24.126597 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.126493    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"
Sep 15 12:54:24.135042 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.135023    2579 scope.go:117] "RemoveContainer" containerID="6c2714d9d092fd3aebad4d154f9fb1827c03f5a5e53d4565f70c9c49e84d5cca"
Sep 15 12:54:24.143645 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.143629    2579 scope.go:117] "RemoveContainer" containerID="56395ca921ddc6b3ebf7bd1f4d38236ae9fd82a53a897ea53e9310f181037b31"
Sep 15 12:54:24.146957 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.146936    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"]
Sep 15 12:54:24.153880 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.152916    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/premium-w4-ee96ec-kserve-7c88698c67-sg8ch"]
Sep 15 12:54:24.166237 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.166214    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"]
Sep 15 12:54:24.169572 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.169549    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/simulator-w4-ee96ec-kserve-779b65bbc4-ssq86"]
Sep 15 12:54:24.178410 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.178391    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"]
Sep 15 12:54:24.182461 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.182437    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ee96ec/distinct-w4-ee96ec-kserve-66cdcd7d5d-smnkp"]
Sep 15 12:54:24.790223 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.790188    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:24.893280 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:24.893236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:25.044302 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:25.044207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:25.104961 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:25.104936    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2baeee9d-c5c1-4ced-8df0-14c60444bfe4" path="/var/lib/kubelet/pods/2baeee9d-c5c1-4ced-8df0-14c60444bfe4/volumes"
Sep 15 12:54:25.105361 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:25.105347    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a45c3a4-8f63-4f88-ae6f-80d2ee45d492" path="/var/lib/kubelet/pods/9a45c3a4-8f63-4f88-ae6f-80d2ee45d492/volumes"
Sep 15 12:54:25.105728 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:25.105715    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bd21233-aa3a-48d6-8e3e-5ecc3fe79170" path="/var/lib/kubelet/pods/9bd21233-aa3a-48d6-8e3e-5ecc3fe79170/volumes"
Sep 15 12:54:26.259300 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:26.259256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:27.447201 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:27.447159    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:54:29.362658 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:29.362627    2579 scope.go:117] "RemoveContainer" containerID="bd716cac22af1f2392c23d56a990c4e61e60dcbb7308c4d7690450607013d5af"
Sep 15 12:54:36.374713 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.374677    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7710-12b31d"]
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375026    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc4ff18-d807-4c99-a70f-76e850dbd59a" containerName="maas-api"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375038    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc4ff18-d807-4c99-a70f-76e850dbd59a" containerName="maas-api"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375049    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bd21233-aa3a-48d6-8e3e-5ecc3fe79170" containerName="main"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375055    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bd21233-aa3a-48d6-8e3e-5ecc3fe79170" containerName="main"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375064    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef085a68-d4f8-466f-b7dc-2d181289f25e" containerName="main"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375071    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef085a68-d4f8-466f-b7dc-2d181289f25e" containerName="main"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375080    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4123e2f2-87d1-41be-a62c-2d2854e52823" containerName="maas-api"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375085    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4123e2f2-87d1-41be-a62c-2d2854e52823" containerName="maas-api"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375098    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d6d2376-d742-421c-bccf-0fcc689eb29b" containerName="main"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375104    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d6d2376-d742-421c-bccf-0fcc689eb29b" containerName="main"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375110    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afd23e0f-7668-41d6-8f66-a8bffa2cc1f2" containerName="maas-api"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375115    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="afd23e0f-7668-41d6-8f66-a8bffa2cc1f2" containerName="maas-api"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375124    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a45c3a4-8f63-4f88-ae6f-80d2ee45d492" containerName="main"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375129    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a45c3a4-8f63-4f88-ae6f-80d2ee45d492" containerName="main"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375135    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ba55e7-ef19-4ade-935a-d4d0ed246c3a" containerName="main"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375140    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ba55e7-ef19-4ade-935a-d4d0ed246c3a" containerName="main"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375146    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ccd6bb-d7db-461f-be67-a6f7312cb5b1" containerName="main"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375151    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ccd6bb-d7db-461f-be67-a6f7312cb5b1" containerName="main"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375158    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2baeee9d-c5c1-4ced-8df0-14c60444bfe4" containerName="main"
Sep 15 12:54:36.375181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375162    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2baeee9d-c5c1-4ced-8df0-14c60444bfe4" containerName="main"
Sep 15 12:54:36.375781 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375226    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef085a68-d4f8-466f-b7dc-2d181289f25e" containerName="main"
Sep 15 12:54:36.375781 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375233    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2baeee9d-c5c1-4ced-8df0-14c60444bfe4" containerName="main"
Sep 15 12:54:36.375781 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375240    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d6d2376-d742-421c-bccf-0fcc689eb29b" containerName="main"
Sep 15 12:54:36.375781 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375246    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ccd6bb-d7db-461f-be67-a6f7312cb5b1" containerName="main"
Sep 15 12:54:36.375781 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375252    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bd21233-aa3a-48d6-8e3e-5ecc3fe79170" containerName="main"
Sep 15 12:54:36.375781 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375256    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="afd23e0f-7668-41d6-8f66-a8bffa2cc1f2" containerName="maas-api"
Sep 15 12:54:36.375781 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375261    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4123e2f2-87d1-41be-a62c-2d2854e52823" containerName="maas-api"
Sep 15 12:54:36.375781 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375267    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ba55e7-ef19-4ade-935a-d4d0ed246c3a" containerName="main"
Sep 15 12:54:36.375781 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375272    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc4ff18-d807-4c99-a70f-76e850dbd59a" containerName="maas-api"
Sep 15 12:54:36.375781 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.375277    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a45c3a4-8f63-4f88-ae6f-80d2ee45d492" containerName="main"
Sep 15 12:54:36.378407 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.378388    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-12b31d"
Sep 15 12:54:36.380810 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.380783    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dhqjr\""
Sep 15 12:54:36.381554 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.381530    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7710-12b31d"]
Sep 15 12:54:36.417875 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.417843    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9z7\" (UniqueName: \"kubernetes.io/projected/50d10d97-4ef7-4a2a-9353-35ec1c693923-kube-api-access-4t9z7\") pod \"test-curl-7710-12b31d\" (UID: \"50d10d97-4ef7-4a2a-9353-35ec1c693923\") " pod="opendatahub/test-curl-7710-12b31d"
Sep 15 12:54:36.519179 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.519146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9z7\" (UniqueName: \"kubernetes.io/projected/50d10d97-4ef7-4a2a-9353-35ec1c693923-kube-api-access-4t9z7\") pod \"test-curl-7710-12b31d\" (UID: \"50d10d97-4ef7-4a2a-9353-35ec1c693923\") " pod="opendatahub/test-curl-7710-12b31d"
Sep 15 12:54:36.526371 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.526344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9z7\" (UniqueName: \"kubernetes.io/projected/50d10d97-4ef7-4a2a-9353-35ec1c693923-kube-api-access-4t9z7\") pod \"test-curl-7710-12b31d\" (UID: \"50d10d97-4ef7-4a2a-9353-35ec1c693923\") " pod="opendatahub/test-curl-7710-12b31d"
Sep 15 12:54:36.689112 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.689026    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-12b31d"
Sep 15 12:54:36.954578 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:36.954548    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7710-12b31d"]
Sep 15 12:54:36.957273 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:54:36.957239    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50d10d97_4ef7_4a2a_9353_35ec1c693923.slice/crio-573a8e228555e48eacca4e55c29dec92bafccf541c0f344af593160d5f54624f WatchSource:0}: Error finding container 573a8e228555e48eacca4e55c29dec92bafccf541c0f344af593160d5f54624f: Status 404 returned error can't find the container with id 573a8e228555e48eacca4e55c29dec92bafccf541c0f344af593160d5f54624f
Sep 15 12:54:37.207586 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:37.207496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7710-12b31d" event={"ID":"50d10d97-4ef7-4a2a-9353-35ec1c693923","Type":"ContainerStarted","Data":"e8d5f0e27a3693d72f18b126261936ec6409c5a93e6f7c3ead4659a2586d99bc"}
Sep 15 12:54:37.207586 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:37.207538    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7710-12b31d" event={"ID":"50d10d97-4ef7-4a2a-9353-35ec1c693923","Type":"ContainerStarted","Data":"573a8e228555e48eacca4e55c29dec92bafccf541c0f344af593160d5f54624f"}
Sep 15 12:54:37.238081 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:37.238029    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7710-12b31d" podStartSLOduration=1.238009519 podStartE2EDuration="1.238009519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:54:37.235173261 +0000 UTC m=+1748.619884144" watchObservedRunningTime="2026-09-15 12:54:37.238009519 +0000 UTC m=+1748.622720403"
Sep 15 12:54:37.814454 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:37.814412    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7710-12b31d"]
Sep 15 12:54:37.819770 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:37.819710    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7710-12b31d"]
Sep 15 12:54:38.212129 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:54:38.212098    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7710-12b31d" container="test-curl-7710-12b31d"
Sep 15 12:54:39.146383 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:39.146349    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"]
Sep 15 12:54:39.165502 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:39.165471    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:39.220197 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:39.220161    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7710-12b31d" podUID="50d10d97-4ef7-4a2a-9353-35ec1c693923" containerName="test-curl-7710-12b31d" containerID="cri-o://e8d5f0e27a3693d72f18b126261936ec6409c5a93e6f7c3ead4659a2586d99bc" gracePeriod=2
Sep 15 12:54:40.351899 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:40.351870    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:41.378903 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:41.378877    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-12b31d"
Sep 15 12:54:41.463197 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:41.463164    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t9z7\" (UniqueName: \"kubernetes.io/projected/50d10d97-4ef7-4a2a-9353-35ec1c693923-kube-api-access-4t9z7\") pod \"50d10d97-4ef7-4a2a-9353-35ec1c693923\" (UID: \"50d10d97-4ef7-4a2a-9353-35ec1c693923\") "
Sep 15 12:54:41.465173 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:41.465139    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d10d97-4ef7-4a2a-9353-35ec1c693923-kube-api-access-4t9z7" (OuterVolumeSpecName: "kube-api-access-4t9z7") pod "50d10d97-4ef7-4a2a-9353-35ec1c693923" (UID: "50d10d97-4ef7-4a2a-9353-35ec1c693923"). InnerVolumeSpecName "kube-api-access-4t9z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:54:41.564892 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:41.564789    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9z7\" (UniqueName: \"kubernetes.io/projected/50d10d97-4ef7-4a2a-9353-35ec1c693923-kube-api-access-4t9z7\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:42.237359 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:42.237328    2579 generic.go:356] "Generic (PLEG): container finished" podID="50d10d97-4ef7-4a2a-9353-35ec1c693923" containerID="e8d5f0e27a3693d72f18b126261936ec6409c5a93e6f7c3ead4659a2586d99bc" exitCode=137
Sep 15 12:54:42.237530 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:42.237372    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-12b31d"
Sep 15 12:54:42.237530 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:42.237404    2579 scope.go:117] "RemoveContainer" containerID="e8d5f0e27a3693d72f18b126261936ec6409c5a93e6f7c3ead4659a2586d99bc"
Sep 15 12:54:42.246486 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:42.246469    2579 scope.go:117] "RemoveContainer" containerID="e8d5f0e27a3693d72f18b126261936ec6409c5a93e6f7c3ead4659a2586d99bc"
Sep 15 12:54:42.246734 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:54:42.246716    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d5f0e27a3693d72f18b126261936ec6409c5a93e6f7c3ead4659a2586d99bc\": container with ID starting with e8d5f0e27a3693d72f18b126261936ec6409c5a93e6f7c3ead4659a2586d99bc not found: ID does not exist" containerID="e8d5f0e27a3693d72f18b126261936ec6409c5a93e6f7c3ead4659a2586d99bc"
Sep 15 12:54:42.246794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:42.246742    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d5f0e27a3693d72f18b126261936ec6409c5a93e6f7c3ead4659a2586d99bc"} err="failed to get container status \"e8d5f0e27a3693d72f18b126261936ec6409c5a93e6f7c3ead4659a2586d99bc\": rpc error: code = NotFound desc = could not find container \"e8d5f0e27a3693d72f18b126261936ec6409c5a93e6f7c3ead4659a2586d99bc\": container with ID starting with e8d5f0e27a3693d72f18b126261936ec6409c5a93e6f7c3ead4659a2586d99bc not found: ID does not exist"
Sep 15 12:54:42.247747 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:54:42.247721    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7710-12b31d\" is forbidden: User \"system:node:ip-10-0-129-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-17.ec2.internal' and this object" podUID="50d10d97-4ef7-4a2a-9353-35ec1c693923" pod="opendatahub/test-curl-7710-12b31d"
Sep 15 12:54:42.254074 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:42.254052    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:54:43.106316 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:43.106285    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50d10d97-4ef7-4a2a-9353-35ec1c693923" path="/var/lib/kubelet/pods/50d10d97-4ef7-4a2a-9353-35ec1c693923/volumes"
Sep 15 12:54:45.152047 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:45.152012    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:45.207984 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:45.207948    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:45.500632 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:45.500546    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:46.954244 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:46.954206    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"]
Sep 15 12:54:46.954641 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:46.954596    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d10d97-4ef7-4a2a-9353-35ec1c693923" containerName="test-curl-7710-12b31d"
Sep 15 12:54:46.954641 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:46.954609    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d10d97-4ef7-4a2a-9353-35ec1c693923" containerName="test-curl-7710-12b31d"
Sep 15 12:54:46.954725 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:46.954685    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d10d97-4ef7-4a2a-9353-35ec1c693923" containerName="test-curl-7710-12b31d"
Sep 15 12:54:46.957452 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:46.957431    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:46.959754 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:46.959730    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-29a9e9\"/\"sub-test-model-29a9e9-kserve-self-signed-certs\""
Sep 15 12:54:46.967958 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:46.967934    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"]
Sep 15 12:54:46.979745 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:46.979715    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:47.114838 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.114810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8435cb43-4683-4973-a91c-e71ddd333dc9-tls-certs\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.114997 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.114879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-model-cache\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.114997 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.114906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-tmp-dir\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.114997 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.114944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjj9b\" (UniqueName: \"kubernetes.io/projected/8435cb43-4683-4973-a91c-e71ddd333dc9-kube-api-access-hjj9b\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.115119 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.115010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-home\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.115119 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.115058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-dshm\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.216561 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.216461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8435cb43-4683-4973-a91c-e71ddd333dc9-tls-certs\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.216561 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.216520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-model-cache\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.216561 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.216553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-tmp-dir\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.216874 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.216581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjj9b\" (UniqueName: \"kubernetes.io/projected/8435cb43-4683-4973-a91c-e71ddd333dc9-kube-api-access-hjj9b\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.216874 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.216611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-home\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.216874 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.216663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-dshm\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.217149 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.217121    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-model-cache\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.217149 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.217137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-home\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.217315 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.217268    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-tmp-dir\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.218990 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.218969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-dshm\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.219053 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.219032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8435cb43-4683-4973-a91c-e71ddd333dc9-tls-certs\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.224732 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.224715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjj9b\" (UniqueName: \"kubernetes.io/projected/8435cb43-4683-4973-a91c-e71ddd333dc9-kube-api-access-hjj9b\") pod \"sub-test-model-29a9e9-kserve-698455d6d9-xmrc7\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.267078 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.267043    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:47.537560 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:47.537533    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"]
Sep 15 12:54:47.539518 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:54:47.539489    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8435cb43_4683_4973_a91c_e71ddd333dc9.slice/crio-8b9edd36641cbe25e2ebc51cc7a3744de2dbcadc9a2d259cf5076331775ec43c WatchSource:0}: Error finding container 8b9edd36641cbe25e2ebc51cc7a3744de2dbcadc9a2d259cf5076331775ec43c: Status 404 returned error can't find the container with id 8b9edd36641cbe25e2ebc51cc7a3744de2dbcadc9a2d259cf5076331775ec43c
Sep 15 12:54:48.269381 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:48.269349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7" event={"ID":"8435cb43-4683-4973-a91c-e71ddd333dc9","Type":"ContainerStarted","Data":"0b3bb9391916fa7db0c7839c08d63f6860c3a8202e31f795367e86675a1a8219"}
Sep 15 12:54:48.269757 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:48.269389    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7" event={"ID":"8435cb43-4683-4973-a91c-e71ddd333dc9","Type":"ContainerStarted","Data":"8b9edd36641cbe25e2ebc51cc7a3744de2dbcadc9a2d259cf5076331775ec43c"}
Sep 15 12:54:48.287227 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:48.287166    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7" podStartSLOduration=2.287148934 podStartE2EDuration="2.287148934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:54:48.286293878 +0000 UTC m=+1759.671004762" watchObservedRunningTime="2026-09-15 12:54:48.287148934 +0000 UTC m=+1759.671859818"
Sep 15 12:54:50.964303 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:50.964267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:54:54.165701 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.165651    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd" podUID="21f9fbfd-ebc8-4057-9a41-b842c8818968" containerName="main" containerID="cri-o://5ef150bbc812ba52ef5f7a3e5995f9dec4ced108e3f548e70b24ddedfc053c45" gracePeriod=45
Sep 15 12:54:54.302507 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.302474    2579 generic.go:356] "Generic (PLEG): container finished" podID="21f9fbfd-ebc8-4057-9a41-b842c8818968" containerID="5ef150bbc812ba52ef5f7a3e5995f9dec4ced108e3f548e70b24ddedfc053c45" exitCode=0
Sep 15 12:54:54.302652 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.302508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd" event={"ID":"21f9fbfd-ebc8-4057-9a41-b842c8818968","Type":"ContainerDied","Data":"5ef150bbc812ba52ef5f7a3e5995f9dec4ced108e3f548e70b24ddedfc053c45"}
Sep 15 12:54:54.408627 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.408600    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:54:54.587305 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.587201    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-dshm\") pod \"21f9fbfd-ebc8-4057-9a41-b842c8818968\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") "
Sep 15 12:54:54.587305 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.587300    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21f9fbfd-ebc8-4057-9a41-b842c8818968-tls-certs\") pod \"21f9fbfd-ebc8-4057-9a41-b842c8818968\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") "
Sep 15 12:54:54.587561 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.587341    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-model-cache\") pod \"21f9fbfd-ebc8-4057-9a41-b842c8818968\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") "
Sep 15 12:54:54.587561 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.587369    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-tmp-dir\") pod \"21f9fbfd-ebc8-4057-9a41-b842c8818968\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") "
Sep 15 12:54:54.587561 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.587404    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-home\") pod \"21f9fbfd-ebc8-4057-9a41-b842c8818968\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") "
Sep 15 12:54:54.587561 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.587430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbvhb\" (UniqueName: \"kubernetes.io/projected/21f9fbfd-ebc8-4057-9a41-b842c8818968-kube-api-access-wbvhb\") pod \"21f9fbfd-ebc8-4057-9a41-b842c8818968\" (UID: \"21f9fbfd-ebc8-4057-9a41-b842c8818968\") "
Sep 15 12:54:54.587752 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.587581    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-model-cache" (OuterVolumeSpecName: "model-cache") pod "21f9fbfd-ebc8-4057-9a41-b842c8818968" (UID: "21f9fbfd-ebc8-4057-9a41-b842c8818968"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:54.587752 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.587689    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21f9fbfd-ebc8-4057-9a41-b842c8818968" (UID: "21f9fbfd-ebc8-4057-9a41-b842c8818968"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:54.587752 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.587698    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-home" (OuterVolumeSpecName: "home") pod "21f9fbfd-ebc8-4057-9a41-b842c8818968" (UID: "21f9fbfd-ebc8-4057-9a41-b842c8818968"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:54.587897 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.587759    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:54.589635 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.589602    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21f9fbfd-ebc8-4057-9a41-b842c8818968-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21f9fbfd-ebc8-4057-9a41-b842c8818968" (UID: "21f9fbfd-ebc8-4057-9a41-b842c8818968"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:54:54.589760 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.589673    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-dshm" (OuterVolumeSpecName: "dshm") pod "21f9fbfd-ebc8-4057-9a41-b842c8818968" (UID: "21f9fbfd-ebc8-4057-9a41-b842c8818968"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:54:54.589760 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.589693    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f9fbfd-ebc8-4057-9a41-b842c8818968-kube-api-access-wbvhb" (OuterVolumeSpecName: "kube-api-access-wbvhb") pod "21f9fbfd-ebc8-4057-9a41-b842c8818968" (UID: "21f9fbfd-ebc8-4057-9a41-b842c8818968"). InnerVolumeSpecName "kube-api-access-wbvhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:54:54.688806 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.688758    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21f9fbfd-ebc8-4057-9a41-b842c8818968-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:54.688806 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.688795    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:54.688806 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.688806    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:54.689073 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.688819    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbvhb\" (UniqueName: \"kubernetes.io/projected/21f9fbfd-ebc8-4057-9a41-b842c8818968-kube-api-access-wbvhb\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:54.689073 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:54.688854    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21f9fbfd-ebc8-4057-9a41-b842c8818968-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:54:55.308950 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:55.308915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd" event={"ID":"21f9fbfd-ebc8-4057-9a41-b842c8818968","Type":"ContainerDied","Data":"34721edb24c87d1e547538fda521297f3f14bf7602c3e2a0e9bd60cb2ca6882e"}
Sep 15 12:54:55.309359 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:55.308965    2579 scope.go:117] "RemoveContainer" containerID="5ef150bbc812ba52ef5f7a3e5995f9dec4ced108e3f548e70b24ddedfc053c45"
Sep 15 12:54:55.309359 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:55.308964    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"
Sep 15 12:54:55.325799 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:55.325772    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"]
Sep 15 12:54:55.329318 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:55.329294    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/auth-test-model-29a9e9-kserve-58685c778c-m9qkd"]
Sep 15 12:54:57.105232 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:57.105200    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21f9fbfd-ebc8-4057-9a41-b842c8818968" path="/var/lib/kubelet/pods/21f9fbfd-ebc8-4057-9a41-b842c8818968/volumes"
Sep 15 12:54:57.267969 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:57.267941    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:57.268132 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:57.267980    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:57.280535 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:57.280511    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:57.330892 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:57.330862    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:54:57.757774 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:54:57.757740    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:55:10.643751 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.643715    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"]
Sep 15 12:55:10.644181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.644104    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21f9fbfd-ebc8-4057-9a41-b842c8818968" containerName="main"
Sep 15 12:55:10.644181 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.644116    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="21f9fbfd-ebc8-4057-9a41-b842c8818968" containerName="main"
Sep 15 12:55:10.644253 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.644187    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="21f9fbfd-ebc8-4057-9a41-b842c8818968" containerName="main"
Sep 15 12:55:10.647357 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.647329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.650010 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.649984    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-17f8f8\"/\"kube-root-ca.crt\""
Sep 15 12:55:10.651248 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.651227    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-17f8f8\"/\"openshift-service-ca.crt\""
Sep 15 12:55:10.651351 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.651244    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-17f8f8\"/\"default-dockercfg-jhmjz\""
Sep 15 12:55:10.651351 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.651304    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-17f8f8\"/\"sub-test-model-17f8f8-kserve-self-signed-certs\""
Sep 15 12:55:10.657809 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.657788    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"]
Sep 15 12:55:10.669351 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.669318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:55:10.716279 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.716247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-tmp-dir\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.716452 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.716293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-model-cache\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.716452 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.716323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-home\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.716452 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.716361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-tls-certs\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.716452 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.716395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvt4\" (UniqueName: \"kubernetes.io/projected/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-kube-api-access-nvvt4\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.716452 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.716420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-dshm\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.816961 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.816913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-tmp-dir\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.816961 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.816975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-model-cache\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.817206 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.816994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-home\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.817206 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.817022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-tls-certs\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.817206 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.817047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvt4\" (UniqueName: \"kubernetes.io/projected/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-kube-api-access-nvvt4\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.817206 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.817067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-dshm\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.817435 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.817405    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-tmp-dir\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.817502 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.817481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-home\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.817565 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.817481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-model-cache\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.819305 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.819285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-dshm\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.819582 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.819564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-tls-certs\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.825082 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.825060    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvt4\" (UniqueName: \"kubernetes.io/projected/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-kube-api-access-nvvt4\") pod \"sub-test-model-17f8f8-kserve-55d65888f9-kbl7z\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") " pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:10.957536 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:10.957431    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:11.228447 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:11.228416    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"]
Sep 15 12:55:11.229641 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:55:11.229610    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc2e0aa5_0ee1_4090_bec4_84afaeda84c2.slice/crio-b29e4b4428284d4359baacd7a7c66842ba95f4674f9b8dacf782e44b66d2c413 WatchSource:0}: Error finding container b29e4b4428284d4359baacd7a7c66842ba95f4674f9b8dacf782e44b66d2c413: Status 404 returned error can't find the container with id b29e4b4428284d4359baacd7a7c66842ba95f4674f9b8dacf782e44b66d2c413
Sep 15 12:55:11.393290 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:11.393255    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z" event={"ID":"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2","Type":"ContainerStarted","Data":"8449b2125b62882e3f31f990ada6539c6f8bf90b53812fd52b8c645952dde2ce"}
Sep 15 12:55:11.393470 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:11.393302    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z" event={"ID":"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2","Type":"ContainerStarted","Data":"b29e4b4428284d4359baacd7a7c66842ba95f4674f9b8dacf782e44b66d2c413"}
Sep 15 12:55:11.408601 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:11.408551    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z" podStartSLOduration=1.408534711 podStartE2EDuration="1.408534711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:55:11.40812343 +0000 UTC m=+1782.792834315" watchObservedRunningTime="2026-09-15 12:55:11.408534711 +0000 UTC m=+1782.793245596"
Sep 15 12:55:14.155742 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:14.155708    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:55:20.958529 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:20.958482    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:20.959002 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:20.958543    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:20.971148 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:20.971126    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:21.455613 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:21.455580    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:55:29.254153 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:29.254129    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:55:29.255666 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:29.255644    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 12:55:33.655392 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:33.655353    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:55:47.147861 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:47.147812    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:55:58.416890 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:58.416850    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7710-15299a"]
Sep 15 12:55:58.420324 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:58.420307    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-15299a"
Sep 15 12:55:58.423089 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:58.423061    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dhqjr\""
Sep 15 12:55:58.426648 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:58.426625    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7710-15299a"]
Sep 15 12:55:58.446732 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:58.446705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8g7g\" (UniqueName: \"kubernetes.io/projected/722790ba-7dc1-4ec1-9753-4684859a589a-kube-api-access-w8g7g\") pod \"test-curl-7710-15299a\" (UID: \"722790ba-7dc1-4ec1-9753-4684859a589a\") " pod="opendatahub/test-curl-7710-15299a"
Sep 15 12:55:58.547622 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:58.547590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8g7g\" (UniqueName: \"kubernetes.io/projected/722790ba-7dc1-4ec1-9753-4684859a589a-kube-api-access-w8g7g\") pod \"test-curl-7710-15299a\" (UID: \"722790ba-7dc1-4ec1-9753-4684859a589a\") " pod="opendatahub/test-curl-7710-15299a"
Sep 15 12:55:58.555525 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:58.555503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8g7g\" (UniqueName: \"kubernetes.io/projected/722790ba-7dc1-4ec1-9753-4684859a589a-kube-api-access-w8g7g\") pod \"test-curl-7710-15299a\" (UID: \"722790ba-7dc1-4ec1-9753-4684859a589a\") " pod="opendatahub/test-curl-7710-15299a"
Sep 15 12:55:58.735650 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:58.735554    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-15299a"
Sep 15 12:55:59.005958 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:59.005926    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7710-15299a"]
Sep 15 12:55:59.007331 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:55:59.007304    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod722790ba_7dc1_4ec1_9753_4684859a589a.slice/crio-175f620d5406935b71f5d50c660038c14a3040dd2af06171dd5e692611537c57 WatchSource:0}: Error finding container 175f620d5406935b71f5d50c660038c14a3040dd2af06171dd5e692611537c57: Status 404 returned error can't find the container with id 175f620d5406935b71f5d50c660038c14a3040dd2af06171dd5e692611537c57
Sep 15 12:55:59.643256 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:59.643216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7710-15299a" event={"ID":"722790ba-7dc1-4ec1-9753-4684859a589a","Type":"ContainerStarted","Data":"e034cab62f4452270405d740ab6487a09c7b99d7f8731b9f31360ec2eeaf55ea"}
Sep 15 12:55:59.643256 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:59.643261    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7710-15299a" event={"ID":"722790ba-7dc1-4ec1-9753-4684859a589a","Type":"ContainerStarted","Data":"175f620d5406935b71f5d50c660038c14a3040dd2af06171dd5e692611537c57"}
Sep 15 12:55:59.657635 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:55:59.657587    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7710-15299a" podStartSLOduration=1.6575727850000002 podStartE2EDuration="1.657572785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:55:59.65650678 +0000 UTC m=+1831.041217665" watchObservedRunningTime="2026-09-15 12:55:59.657572785 +0000 UTC m=+1831.042283669"
Sep 15 12:56:00.311021 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:00.310979    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7710-15299a"]
Sep 15 12:56:00.314863 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:00.314819    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7710-15299a"]
Sep 15 12:56:00.649092 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:56:00.649062    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7710-15299a" container="test-curl-7710-15299a"
Sep 15 12:56:01.654420 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:01.654356    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7710-15299a" podUID="722790ba-7dc1-4ec1-9753-4684859a589a" containerName="test-curl-7710-15299a" containerID="cri-o://e034cab62f4452270405d740ab6487a09c7b99d7f8731b9f31360ec2eeaf55ea" gracePeriod=2
Sep 15 12:56:02.322815 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:02.322779    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"]
Sep 15 12:56:02.343096 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:02.343065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:56:03.243430 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:03.243392    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"]
Sep 15 12:56:03.538419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:03.538323    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:56:03.844187 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:03.844165    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-15299a"
Sep 15 12:56:03.892959 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:03.892929    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8g7g\" (UniqueName: \"kubernetes.io/projected/722790ba-7dc1-4ec1-9753-4684859a589a-kube-api-access-w8g7g\") pod \"722790ba-7dc1-4ec1-9753-4684859a589a\" (UID: \"722790ba-7dc1-4ec1-9753-4684859a589a\") "
Sep 15 12:56:03.895016 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:03.894993    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722790ba-7dc1-4ec1-9753-4684859a589a-kube-api-access-w8g7g" (OuterVolumeSpecName: "kube-api-access-w8g7g") pod "722790ba-7dc1-4ec1-9753-4684859a589a" (UID: "722790ba-7dc1-4ec1-9753-4684859a589a"). InnerVolumeSpecName "kube-api-access-w8g7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:56:03.993896 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:03.993859    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8g7g\" (UniqueName: \"kubernetes.io/projected/722790ba-7dc1-4ec1-9753-4684859a589a-kube-api-access-w8g7g\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:56:04.672164 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:04.672124    2579 generic.go:356] "Generic (PLEG): container finished" podID="722790ba-7dc1-4ec1-9753-4684859a589a" containerID="e034cab62f4452270405d740ab6487a09c7b99d7f8731b9f31360ec2eeaf55ea" exitCode=137
Sep 15 12:56:04.672164 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:04.672167    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-15299a"
Sep 15 12:56:04.672751 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:04.672214    2579 scope.go:117] "RemoveContainer" containerID="e034cab62f4452270405d740ab6487a09c7b99d7f8731b9f31360ec2eeaf55ea"
Sep 15 12:56:04.681081 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:04.681063    2579 scope.go:117] "RemoveContainer" containerID="e034cab62f4452270405d740ab6487a09c7b99d7f8731b9f31360ec2eeaf55ea"
Sep 15 12:56:04.681388 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:56:04.681368    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e034cab62f4452270405d740ab6487a09c7b99d7f8731b9f31360ec2eeaf55ea\": container with ID starting with e034cab62f4452270405d740ab6487a09c7b99d7f8731b9f31360ec2eeaf55ea not found: ID does not exist" containerID="e034cab62f4452270405d740ab6487a09c7b99d7f8731b9f31360ec2eeaf55ea"
Sep 15 12:56:04.681458 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:04.681402    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e034cab62f4452270405d740ab6487a09c7b99d7f8731b9f31360ec2eeaf55ea"} err="failed to get container status \"e034cab62f4452270405d740ab6487a09c7b99d7f8731b9f31360ec2eeaf55ea\": rpc error: code = NotFound desc = could not find container \"e034cab62f4452270405d740ab6487a09c7b99d7f8731b9f31360ec2eeaf55ea\": container with ID starting with e034cab62f4452270405d740ab6487a09c7b99d7f8731b9f31360ec2eeaf55ea not found: ID does not exist"
Sep 15 12:56:04.682254 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:56:04.682229    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7710-15299a\" is forbidden: User \"system:node:ip-10-0-129-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-17.ec2.internal' and this object" podUID="722790ba-7dc1-4ec1-9753-4684859a589a" pod="opendatahub/test-curl-7710-15299a"
Sep 15 12:56:05.105718 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:05.105691    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="722790ba-7dc1-4ec1-9753-4684859a589a" path="/var/lib/kubelet/pods/722790ba-7dc1-4ec1-9753-4684859a589a/volumes"
Sep 15 12:56:05.555814 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:05.555736    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:56:06.276572 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:06.276541    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:56:17.341384 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.341342    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7" podUID="8435cb43-4683-4973-a91c-e71ddd333dc9" containerName="main" containerID="cri-o://0b3bb9391916fa7db0c7839c08d63f6860c3a8202e31f795367e86675a1a8219" gracePeriod=45
Sep 15 12:56:17.580556 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.580532    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:56:17.722222 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.722191    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-home\") pod \"8435cb43-4683-4973-a91c-e71ddd333dc9\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") "
Sep 15 12:56:17.722415 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.722236    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-tmp-dir\") pod \"8435cb43-4683-4973-a91c-e71ddd333dc9\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") "
Sep 15 12:56:17.722415 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.722263    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-model-cache\") pod \"8435cb43-4683-4973-a91c-e71ddd333dc9\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") "
Sep 15 12:56:17.722415 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.722290    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-dshm\") pod \"8435cb43-4683-4973-a91c-e71ddd333dc9\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") "
Sep 15 12:56:17.722415 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.722381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8435cb43-4683-4973-a91c-e71ddd333dc9-tls-certs\") pod \"8435cb43-4683-4973-a91c-e71ddd333dc9\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") "
Sep 15 12:56:17.722415 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.722408    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjj9b\" (UniqueName: \"kubernetes.io/projected/8435cb43-4683-4973-a91c-e71ddd333dc9-kube-api-access-hjj9b\") pod \"8435cb43-4683-4973-a91c-e71ddd333dc9\" (UID: \"8435cb43-4683-4973-a91c-e71ddd333dc9\") "
Sep 15 12:56:17.722670 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.722512    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8435cb43-4683-4973-a91c-e71ddd333dc9" (UID: "8435cb43-4683-4973-a91c-e71ddd333dc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:56:17.722670 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.722530    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-home" (OuterVolumeSpecName: "home") pod "8435cb43-4683-4973-a91c-e71ddd333dc9" (UID: "8435cb43-4683-4973-a91c-e71ddd333dc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:56:17.722670 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.722543    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-model-cache" (OuterVolumeSpecName: "model-cache") pod "8435cb43-4683-4973-a91c-e71ddd333dc9" (UID: "8435cb43-4683-4973-a91c-e71ddd333dc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:56:17.722799 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.722671    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:56:17.722799 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.722682    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:56:17.722799 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.722692    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:56:17.724501 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.724477    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-dshm" (OuterVolumeSpecName: "dshm") pod "8435cb43-4683-4973-a91c-e71ddd333dc9" (UID: "8435cb43-4683-4973-a91c-e71ddd333dc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:56:17.724610 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.724515    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8435cb43-4683-4973-a91c-e71ddd333dc9-kube-api-access-hjj9b" (OuterVolumeSpecName: "kube-api-access-hjj9b") pod "8435cb43-4683-4973-a91c-e71ddd333dc9" (UID: "8435cb43-4683-4973-a91c-e71ddd333dc9"). InnerVolumeSpecName "kube-api-access-hjj9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:56:17.724610 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.724531    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8435cb43-4683-4973-a91c-e71ddd333dc9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8435cb43-4683-4973-a91c-e71ddd333dc9" (UID: "8435cb43-4683-4973-a91c-e71ddd333dc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:56:17.748502 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.748472    2579 generic.go:356] "Generic (PLEG): container finished" podID="8435cb43-4683-4973-a91c-e71ddd333dc9" containerID="0b3bb9391916fa7db0c7839c08d63f6860c3a8202e31f795367e86675a1a8219" exitCode=0
Sep 15 12:56:17.748626 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.748529    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"
Sep 15 12:56:17.748626 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.748554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7" event={"ID":"8435cb43-4683-4973-a91c-e71ddd333dc9","Type":"ContainerDied","Data":"0b3bb9391916fa7db0c7839c08d63f6860c3a8202e31f795367e86675a1a8219"}
Sep 15 12:56:17.748626 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.748598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7" event={"ID":"8435cb43-4683-4973-a91c-e71ddd333dc9","Type":"ContainerDied","Data":"8b9edd36641cbe25e2ebc51cc7a3744de2dbcadc9a2d259cf5076331775ec43c"}
Sep 15 12:56:17.748626 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.748619    2579 scope.go:117] "RemoveContainer" containerID="0b3bb9391916fa7db0c7839c08d63f6860c3a8202e31f795367e86675a1a8219"
Sep 15 12:56:17.758538 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.758521    2579 scope.go:117] "RemoveContainer" containerID="0b3bb9391916fa7db0c7839c08d63f6860c3a8202e31f795367e86675a1a8219"
Sep 15 12:56:17.758875 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:56:17.758849    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b3bb9391916fa7db0c7839c08d63f6860c3a8202e31f795367e86675a1a8219\": container with ID starting with 0b3bb9391916fa7db0c7839c08d63f6860c3a8202e31f795367e86675a1a8219 not found: ID does not exist" containerID="0b3bb9391916fa7db0c7839c08d63f6860c3a8202e31f795367e86675a1a8219"
Sep 15 12:56:17.758953 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.758888    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b3bb9391916fa7db0c7839c08d63f6860c3a8202e31f795367e86675a1a8219"} err="failed to get container status \"0b3bb9391916fa7db0c7839c08d63f6860c3a8202e31f795367e86675a1a8219\": rpc error: code = NotFound desc = could not find container \"0b3bb9391916fa7db0c7839c08d63f6860c3a8202e31f795367e86675a1a8219\": container with ID starting with 0b3bb9391916fa7db0c7839c08d63f6860c3a8202e31f795367e86675a1a8219 not found: ID does not exist"
Sep 15 12:56:17.772098 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.772042    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"]
Sep 15 12:56:17.774659 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.774634    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/sub-test-model-29a9e9-kserve-698455d6d9-xmrc7"]
Sep 15 12:56:17.823596 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.823566    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8435cb43-4683-4973-a91c-e71ddd333dc9-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:56:17.823596 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.823590    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjj9b\" (UniqueName: \"kubernetes.io/projected/8435cb43-4683-4973-a91c-e71ddd333dc9-kube-api-access-hjj9b\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:56:17.823596 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:17.823599    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8435cb43-4683-4973-a91c-e71ddd333dc9-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:56:18.262440 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.262372    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z" podUID="cc2e0aa5-0ee1-4090-bec4-84afaeda84c2" containerName="main" containerID="cri-o://8449b2125b62882e3f31f990ada6539c6f8bf90b53812fd52b8c645952dde2ce" gracePeriod=45
Sep 15 12:56:18.510787 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.510763    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:56:18.631394 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.631359    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-home\") pod \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") "
Sep 15 12:56:18.631604 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.631422    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvvt4\" (UniqueName: \"kubernetes.io/projected/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-kube-api-access-nvvt4\") pod \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") "
Sep 15 12:56:18.631604 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.631443    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-model-cache\") pod \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") "
Sep 15 12:56:18.631604 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.631472    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-dshm\") pod \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") "
Sep 15 12:56:18.631604 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.631524    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-tls-certs\") pod \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") "
Sep 15 12:56:18.631809 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.631654    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-tmp-dir\") pod \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\" (UID: \"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2\") "
Sep 15 12:56:18.631809 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.631661    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-home" (OuterVolumeSpecName: "home") pod "cc2e0aa5-0ee1-4090-bec4-84afaeda84c2" (UID: "cc2e0aa5-0ee1-4090-bec4-84afaeda84c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:56:18.631809 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.631771    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc2e0aa5-0ee1-4090-bec4-84afaeda84c2" (UID: "cc2e0aa5-0ee1-4090-bec4-84afaeda84c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:56:18.631991 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.631901    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc2e0aa5-0ee1-4090-bec4-84afaeda84c2" (UID: "cc2e0aa5-0ee1-4090-bec4-84afaeda84c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:56:18.632056 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.632038    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:56:18.632096 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.632058    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:56:18.632096 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.632067    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:56:18.633845 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.633803    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-kube-api-access-nvvt4" (OuterVolumeSpecName: "kube-api-access-nvvt4") pod "cc2e0aa5-0ee1-4090-bec4-84afaeda84c2" (UID: "cc2e0aa5-0ee1-4090-bec4-84afaeda84c2"). InnerVolumeSpecName "kube-api-access-nvvt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:56:18.633959 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.633853    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc2e0aa5-0ee1-4090-bec4-84afaeda84c2" (UID: "cc2e0aa5-0ee1-4090-bec4-84afaeda84c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:56:18.633959 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.633864    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-dshm" (OuterVolumeSpecName: "dshm") pod "cc2e0aa5-0ee1-4090-bec4-84afaeda84c2" (UID: "cc2e0aa5-0ee1-4090-bec4-84afaeda84c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:56:18.733099 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.733064    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:56:18.733099 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.733095    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvvt4\" (UniqueName: \"kubernetes.io/projected/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-kube-api-access-nvvt4\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:56:18.733317 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.733107    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:56:18.755954 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.755921    2579 generic.go:356] "Generic (PLEG): container finished" podID="cc2e0aa5-0ee1-4090-bec4-84afaeda84c2" containerID="8449b2125b62882e3f31f990ada6539c6f8bf90b53812fd52b8c645952dde2ce" exitCode=0
Sep 15 12:56:18.756094 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.755968    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"
Sep 15 12:56:18.756094 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.756004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z" event={"ID":"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2","Type":"ContainerDied","Data":"8449b2125b62882e3f31f990ada6539c6f8bf90b53812fd52b8c645952dde2ce"}
Sep 15 12:56:18.756094 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.756046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z" event={"ID":"cc2e0aa5-0ee1-4090-bec4-84afaeda84c2","Type":"ContainerDied","Data":"b29e4b4428284d4359baacd7a7c66842ba95f4674f9b8dacf782e44b66d2c413"}
Sep 15 12:56:18.756094 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.756064    2579 scope.go:117] "RemoveContainer" containerID="8449b2125b62882e3f31f990ada6539c6f8bf90b53812fd52b8c645952dde2ce"
Sep 15 12:56:18.765374 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.765356    2579 scope.go:117] "RemoveContainer" containerID="8449b2125b62882e3f31f990ada6539c6f8bf90b53812fd52b8c645952dde2ce"
Sep 15 12:56:18.765619 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:56:18.765601    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8449b2125b62882e3f31f990ada6539c6f8bf90b53812fd52b8c645952dde2ce\": container with ID starting with 8449b2125b62882e3f31f990ada6539c6f8bf90b53812fd52b8c645952dde2ce not found: ID does not exist" containerID="8449b2125b62882e3f31f990ada6539c6f8bf90b53812fd52b8c645952dde2ce"
Sep 15 12:56:18.765666 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.765624    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8449b2125b62882e3f31f990ada6539c6f8bf90b53812fd52b8c645952dde2ce"} err="failed to get container status \"8449b2125b62882e3f31f990ada6539c6f8bf90b53812fd52b8c645952dde2ce\": rpc error: code = NotFound desc = could not find container \"8449b2125b62882e3f31f990ada6539c6f8bf90b53812fd52b8c645952dde2ce\": container with ID starting with 8449b2125b62882e3f31f990ada6539c6f8bf90b53812fd52b8c645952dde2ce not found: ID does not exist"
Sep 15 12:56:18.778254 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.778233    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"]
Sep 15 12:56:18.782372 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:18.782351    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-17f8f8/sub-test-model-17f8f8-kserve-55d65888f9-kbl7z"]
Sep 15 12:56:19.105659 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:19.105632    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8435cb43-4683-4973-a91c-e71ddd333dc9" path="/var/lib/kubelet/pods/8435cb43-4683-4973-a91c-e71ddd333dc9/volumes"
Sep 15 12:56:19.106040 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:19.106027    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc2e0aa5-0ee1-4090-bec4-84afaeda84c2" path="/var/lib/kubelet/pods/cc2e0aa5-0ee1-4090-bec4-84afaeda84c2/volumes"
Sep 15 12:56:20.298086 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:20.298046    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:56:33.113560 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.113528    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"]
Sep 15 12:56:33.114091 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.114072    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8435cb43-4683-4973-a91c-e71ddd333dc9" containerName="main"
Sep 15 12:56:33.114140 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.114096    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8435cb43-4683-4973-a91c-e71ddd333dc9" containerName="main"
Sep 15 12:56:33.114140 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.114107    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2e0aa5-0ee1-4090-bec4-84afaeda84c2" containerName="main"
Sep 15 12:56:33.114140 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.114115    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2e0aa5-0ee1-4090-bec4-84afaeda84c2" containerName="main"
Sep 15 12:56:33.114140 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.114138    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="722790ba-7dc1-4ec1-9753-4684859a589a" containerName="test-curl-7710-15299a"
Sep 15 12:56:33.114268 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.114145    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="722790ba-7dc1-4ec1-9753-4684859a589a" containerName="test-curl-7710-15299a"
Sep 15 12:56:33.114268 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.114210    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8435cb43-4683-4973-a91c-e71ddd333dc9" containerName="main"
Sep 15 12:56:33.114268 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.114217    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2e0aa5-0ee1-4090-bec4-84afaeda84c2" containerName="main"
Sep 15 12:56:33.114268 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.114225    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="722790ba-7dc1-4ec1-9753-4684859a589a" containerName="test-curl-7710-15299a"
Sep 15 12:56:33.117173 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.117156    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.121387 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.121361    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-29a9e9\"/\"default-dockercfg-gwmjv\""
Sep 15 12:56:33.121387 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.121380    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-29a9e9\"/\"rate-test-model-29a9e9-kserve-self-signed-certs\""
Sep 15 12:56:33.121585 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.121422    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-29a9e9\"/\"kube-root-ca.crt\""
Sep 15 12:56:33.121585 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.121456    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-29a9e9\"/\"openshift-service-ca.crt\""
Sep 15 12:56:33.139697 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.139675    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"]
Sep 15 12:56:33.169106 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.169077    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:56:33.259988 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.259956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-tls-certs\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.260141 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.259994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-model-cache\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.260141 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.260112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2ddn\" (UniqueName: \"kubernetes.io/projected/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-kube-api-access-m2ddn\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.260220 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.260154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-dshm\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.260220 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.260173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-tmp-dir\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.260220 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.260205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-home\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.361435 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.361402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2ddn\" (UniqueName: \"kubernetes.io/projected/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-kube-api-access-m2ddn\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.361577 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.361444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-dshm\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.361577 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.361465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-tmp-dir\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.361677 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.361573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-home\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.361677 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.361630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-tls-certs\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.361677 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.361668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-model-cache\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.361872 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.361793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-tmp-dir\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.361965 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.361945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-home\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.362027 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.362004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-model-cache\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.363765 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.363708    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-dshm\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.364016 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.363999    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-tls-certs\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.372114 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.372093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2ddn\" (UniqueName: \"kubernetes.io/projected/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-kube-api-access-m2ddn\") pod \"rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") " pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.426057 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.426012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:33.694627 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.694601    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"]
Sep 15 12:56:33.696410 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:56:33.696369    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod764fa6f1_079f_4bb0_a0ba_d224ef7459ab.slice/crio-49c88cd1e56fa0a863f6b11f6559e3458db553a59ef59effff020578f5884f8c WatchSource:0}: Error finding container 49c88cd1e56fa0a863f6b11f6559e3458db553a59ef59effff020578f5884f8c: Status 404 returned error can't find the container with id 49c88cd1e56fa0a863f6b11f6559e3458db553a59ef59effff020578f5884f8c
Sep 15 12:56:33.832513 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.832474    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb" event={"ID":"764fa6f1-079f-4bb0-a0ba-d224ef7459ab","Type":"ContainerStarted","Data":"3436b4bc4580b35d98141b309dfb8873bed2d4d890c1909d56cdc0833ef2e791"}
Sep 15 12:56:33.832683 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.832529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb" event={"ID":"764fa6f1-079f-4bb0-a0ba-d224ef7459ab","Type":"ContainerStarted","Data":"49c88cd1e56fa0a863f6b11f6559e3458db553a59ef59effff020578f5884f8c"}
Sep 15 12:56:33.850756 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:33.850704    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb" podStartSLOduration=0.850688388 podStartE2EDuration="850.688388ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:56:33.848899711 +0000 UTC m=+1865.233610594" watchObservedRunningTime="2026-09-15 12:56:33.850688388 +0000 UTC m=+1865.235399275"
Sep 15 12:56:36.456937 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:36.456895    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:56:43.426624 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:43.426582    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:43.427150 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:43.426651    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:43.440105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:43.440081    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:43.578964 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:43.578923    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:56:43.680134 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:43.680042    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:56:43.897316 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:43.897287    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:56:47.537275 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:47.537207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:56:51.152286 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:51.152251    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:56:56.089219 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:56:56.089179    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:11.346786 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:11.346750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:57:13.442115 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:13.442076    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:13.950348 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:13.950310    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:20.756893 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:20.756860    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:57:27.734394 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:27.734364    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"]
Sep 15 12:57:27.746551 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:27.746510    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:28.658499 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:28.658464    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:29.083681 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:29.083590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:29.390803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:29.390764    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:30.003220 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:30.003177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:31.956699 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:31.956662    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:57:32.179903 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:32.179858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:32.817690 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:32.817637    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:32.904159 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:32.904117    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:33.771157 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:33.771116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:34.560384 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:34.560356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:34.611864 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:34.611814    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:34.712101 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:34.712070    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:35.022504 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:35.022474    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:35.123106 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:35.123067    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:42.753823 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:42.753786    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb" podUID="764fa6f1-079f-4bb0-a0ba-d224ef7459ab" containerName="main" containerID="cri-o://3436b4bc4580b35d98141b309dfb8873bed2d4d890c1909d56cdc0833ef2e791" gracePeriod=45
Sep 15 12:57:43.002889 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.002866    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:57:43.076416 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.076334    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-dshm\") pod \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") "
Sep 15 12:57:43.076416 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.076368    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-home\") pod \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") "
Sep 15 12:57:43.076416 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.076399    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-model-cache\") pod \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") "
Sep 15 12:57:43.076416 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.076418    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-tls-certs\") pod \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") "
Sep 15 12:57:43.076750 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.076483    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2ddn\" (UniqueName: \"kubernetes.io/projected/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-kube-api-access-m2ddn\") pod \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") "
Sep 15 12:57:43.076750 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.076509    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-tmp-dir\") pod \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\" (UID: \"764fa6f1-079f-4bb0-a0ba-d224ef7459ab\") "
Sep 15 12:57:43.076750 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.076639    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-home" (OuterVolumeSpecName: "home") pod "764fa6f1-079f-4bb0-a0ba-d224ef7459ab" (UID: "764fa6f1-079f-4bb0-a0ba-d224ef7459ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:57:43.076750 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.076659    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "764fa6f1-079f-4bb0-a0ba-d224ef7459ab" (UID: "764fa6f1-079f-4bb0-a0ba-d224ef7459ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:57:43.076750 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.076748    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:57:43.077041 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.076767    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:57:43.077041 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.076862    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "764fa6f1-079f-4bb0-a0ba-d224ef7459ab" (UID: "764fa6f1-079f-4bb0-a0ba-d224ef7459ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:57:43.078471 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.078448    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-dshm" (OuterVolumeSpecName: "dshm") pod "764fa6f1-079f-4bb0-a0ba-d224ef7459ab" (UID: "764fa6f1-079f-4bb0-a0ba-d224ef7459ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:57:43.078569 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.078511    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "764fa6f1-079f-4bb0-a0ba-d224ef7459ab" (UID: "764fa6f1-079f-4bb0-a0ba-d224ef7459ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:57:43.078668 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.078652    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-kube-api-access-m2ddn" (OuterVolumeSpecName: "kube-api-access-m2ddn") pod "764fa6f1-079f-4bb0-a0ba-d224ef7459ab" (UID: "764fa6f1-079f-4bb0-a0ba-d224ef7459ab"). InnerVolumeSpecName "kube-api-access-m2ddn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:57:43.178007 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.177963    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:57:43.178007 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.178000    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2ddn\" (UniqueName: \"kubernetes.io/projected/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-kube-api-access-m2ddn\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:57:43.178007 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.178011    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:57:43.178007 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.178019    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/764fa6f1-079f-4bb0-a0ba-d224ef7459ab-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:57:43.187401 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.187373    2579 generic.go:356] "Generic (PLEG): container finished" podID="764fa6f1-079f-4bb0-a0ba-d224ef7459ab" containerID="3436b4bc4580b35d98141b309dfb8873bed2d4d890c1909d56cdc0833ef2e791" exitCode=0
Sep 15 12:57:43.187526 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.187423    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"
Sep 15 12:57:43.187526 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.187466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb" event={"ID":"764fa6f1-079f-4bb0-a0ba-d224ef7459ab","Type":"ContainerDied","Data":"3436b4bc4580b35d98141b309dfb8873bed2d4d890c1909d56cdc0833ef2e791"}
Sep 15 12:57:43.187526 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.187509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb" event={"ID":"764fa6f1-079f-4bb0-a0ba-d224ef7459ab","Type":"ContainerDied","Data":"49c88cd1e56fa0a863f6b11f6559e3458db553a59ef59effff020578f5884f8c"}
Sep 15 12:57:43.187526 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.187527    2579 scope.go:117] "RemoveContainer" containerID="3436b4bc4580b35d98141b309dfb8873bed2d4d890c1909d56cdc0833ef2e791"
Sep 15 12:57:43.196187 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.196168    2579 scope.go:117] "RemoveContainer" containerID="3436b4bc4580b35d98141b309dfb8873bed2d4d890c1909d56cdc0833ef2e791"
Sep 15 12:57:43.196451 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:57:43.196435    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3436b4bc4580b35d98141b309dfb8873bed2d4d890c1909d56cdc0833ef2e791\": container with ID starting with 3436b4bc4580b35d98141b309dfb8873bed2d4d890c1909d56cdc0833ef2e791 not found: ID does not exist" containerID="3436b4bc4580b35d98141b309dfb8873bed2d4d890c1909d56cdc0833ef2e791"
Sep 15 12:57:43.196497 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.196459    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3436b4bc4580b35d98141b309dfb8873bed2d4d890c1909d56cdc0833ef2e791"} err="failed to get container status \"3436b4bc4580b35d98141b309dfb8873bed2d4d890c1909d56cdc0833ef2e791\": rpc error: code = NotFound desc = could not find container \"3436b4bc4580b35d98141b309dfb8873bed2d4d890c1909d56cdc0833ef2e791\": container with ID starting with 3436b4bc4580b35d98141b309dfb8873bed2d4d890c1909d56cdc0833ef2e791 not found: ID does not exist"
Sep 15 12:57:43.203279 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.203254    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"]
Sep 15 12:57:43.206878 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:43.206856    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-29a9e9/rate-test-model-29a9e9-kserve-576bb9b7f9-6xqcb"]
Sep 15 12:57:44.449670 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:44.449635    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:57:45.105481 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:45.105452    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="764fa6f1-079f-4bb0-a0ba-d224ef7459ab" path="/var/lib/kubelet/pods/764fa6f1-079f-4bb0-a0ba-d224ef7459ab/volumes"
Sep 15 12:57:49.646419 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:49.646385    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:55.394936 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:55.394895    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:56.770949 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:56.770912    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:57.407360 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:57.407318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:57.877970 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:57.877935    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:58.074599 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:58.074562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:58.556632 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:58.556594    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:58.778875 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:58.778822    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:58.858243 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:58.858197    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:57:59.189897 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:57:59.189782    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:01.822711 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:01.822653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:02.018094 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:02.018058    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:02.119061 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:02.119031    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:02.796890 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:02.796851    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:03.479964 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:03.479920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:04.258426 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.258394    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"]
Sep 15 12:58:04.258792 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.258778    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764fa6f1-079f-4bb0-a0ba-d224ef7459ab" containerName="main"
Sep 15 12:58:04.258883 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.258795    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="764fa6f1-079f-4bb0-a0ba-d224ef7459ab" containerName="main"
Sep 15 12:58:04.258954 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.258890    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="764fa6f1-079f-4bb0-a0ba-d224ef7459ab" containerName="main"
Sep 15 12:58:04.261967 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.261942    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:04.264781 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.264760    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-cd72a9-serving-cert\""
Sep 15 12:58:04.271815 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.271789    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"]
Sep 15 12:58:04.291964 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.291934    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:04.360653 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.360622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79efd4dc-69e1-4262-b7d5-20914de57933-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cd72a9-d7474695-msj85\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:04.360807 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.360666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncvww\" (UniqueName: \"kubernetes.io/projected/79efd4dc-69e1-4262-b7d5-20914de57933-kube-api-access-ncvww\") pod \"maas-api-e2e-ipp-b-cd72a9-d7474695-msj85\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:04.360807 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.360695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79efd4dc-69e1-4262-b7d5-20914de57933-metrics-certs\") pod \"maas-api-e2e-ipp-b-cd72a9-d7474695-msj85\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:04.360925 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.360816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79efd4dc-69e1-4262-b7d5-20914de57933-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cd72a9-d7474695-msj85\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:04.393057 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.393030    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:04.461712 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.461680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncvww\" (UniqueName: \"kubernetes.io/projected/79efd4dc-69e1-4262-b7d5-20914de57933-kube-api-access-ncvww\") pod \"maas-api-e2e-ipp-b-cd72a9-d7474695-msj85\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:04.461911 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.461725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79efd4dc-69e1-4262-b7d5-20914de57933-metrics-certs\") pod \"maas-api-e2e-ipp-b-cd72a9-d7474695-msj85\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:04.461911 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.461785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79efd4dc-69e1-4262-b7d5-20914de57933-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cd72a9-d7474695-msj85\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:04.461911 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.461811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79efd4dc-69e1-4262-b7d5-20914de57933-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cd72a9-d7474695-msj85\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:04.462044 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:58:04.461935    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-serving-cert: secret "maas-api-e2e-ipp-b-cd72a9-serving-cert" not found
Sep 15 12:58:04.462044 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:58:04.462010    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79efd4dc-69e1-4262-b7d5-20914de57933-maas-api-tls podName:79efd4dc-69e1-4262-b7d5-20914de57933 nodeName:}" failed. No retries permitted until 2026-09-15 12:58:04.961992102 +0000 UTC m=+1956.346702966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79efd4dc-69e1-4262-b7d5-20914de57933-maas-api-tls") pod "maas-api-e2e-ipp-b-cd72a9-d7474695-msj85" (UID: "79efd4dc-69e1-4262-b7d5-20914de57933") : secret "maas-api-e2e-ipp-b-cd72a9-serving-cert" not found
Sep 15 12:58:04.462478 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.462456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79efd4dc-69e1-4262-b7d5-20914de57933-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cd72a9-d7474695-msj85\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:04.464515 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.464493    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79efd4dc-69e1-4262-b7d5-20914de57933-metrics-certs\") pod \"maas-api-e2e-ipp-b-cd72a9-d7474695-msj85\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:04.472007 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.471987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncvww\" (UniqueName: \"kubernetes.io/projected/79efd4dc-69e1-4262-b7d5-20914de57933-kube-api-access-ncvww\") pod \"maas-api-e2e-ipp-b-cd72a9-d7474695-msj85\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:04.617636 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.617598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:04.735626 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.735583    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:04.836140 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.836108    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:04.967694 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.967613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79efd4dc-69e1-4262-b7d5-20914de57933-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cd72a9-d7474695-msj85\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:04.970099 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:04.970069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79efd4dc-69e1-4262-b7d5-20914de57933-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cd72a9-d7474695-msj85\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:05.175522 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:05.175487    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:05.447309 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:05.447282    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"]
Sep 15 12:58:05.449045 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:58:05.449019    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79efd4dc_69e1_4262_b7d5_20914de57933.slice/crio-c4df3f5a5eb110abff9ce01d3dd7c83e7697dc0dce7c5f15829f3f3f53473f90 WatchSource:0}: Error finding container c4df3f5a5eb110abff9ce01d3dd7c83e7697dc0dce7c5f15829f3f3f53473f90: Status 404 returned error can't find the container with id c4df3f5a5eb110abff9ce01d3dd7c83e7697dc0dce7c5f15829f3f3f53473f90
Sep 15 12:58:05.450417 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:05.450400    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 12:58:06.308850 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:06.308790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85" event={"ID":"79efd4dc-69e1-4262-b7d5-20914de57933","Type":"ContainerStarted","Data":"8a9319756760591e4e01d1dee1fef99c34674d7bb97392e86ad41e5146756b6e"}
Sep 15 12:58:06.309261 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:06.308976    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:06.309261 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:06.309025    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85" event={"ID":"79efd4dc-69e1-4262-b7d5-20914de57933","Type":"ContainerStarted","Data":"c4df3f5a5eb110abff9ce01d3dd7c83e7697dc0dce7c5f15829f3f3f53473f90"}
Sep 15 12:58:06.326061 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:06.326014    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85" podStartSLOduration=1.963250677 podStartE2EDuration="2.32599982s" totalImagesPullingTime="362.749143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:58:06.325432475 +0000 UTC m=+1957.710143360" watchObservedRunningTime="2026-09-15 12:58:06.32599982 +0000 UTC m=+1957.710710704"
Sep 15 12:58:09.747625 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:09.747589    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:58:12.320591 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:12.320561    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 12:58:26.179179 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:26.179082    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:26.813058 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:26.813011    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:27.079861 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:27.079755    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:27.514315 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:27.514271    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:28.287245 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:28.287207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:28.922321 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:28.922283    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:46.937422 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:46.937380    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"]
Sep 15 12:58:46.941233 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:46.941212    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:46.945595 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:46.945571    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f0171b\"/\"default-dockercfg-6w6ms\""
Sep 15 12:58:46.945595 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:46.945593    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f0171b\"/\"openshift-service-ca.crt\""
Sep 15 12:58:46.945860 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:46.945571    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f0171b\"/\"kube-root-ca.crt\""
Sep 15 12:58:46.945860 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:46.945571    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f0171b\"/\"ipp-route-f0171b-kserve-self-signed-certs\""
Sep 15 12:58:46.950723 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:46.950698    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"]
Sep 15 12:58:46.982079 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:46.982051    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:58:47.023315 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.023286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht69d\" (UniqueName: \"kubernetes.io/projected/12dd8566-525c-4af9-84a6-418479e879aa-kube-api-access-ht69d\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.023447 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.023343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-home\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.023447 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.023362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12dd8566-525c-4af9-84a6-418479e879aa-tls-certs\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.023447 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.023382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-dshm\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.023447 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.023402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-model-cache\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.023589 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.023456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-tmp-dir\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.124796 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.124764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht69d\" (UniqueName: \"kubernetes.io/projected/12dd8566-525c-4af9-84a6-418479e879aa-kube-api-access-ht69d\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.125005 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.124816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-home\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.125005 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.124853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12dd8566-525c-4af9-84a6-418479e879aa-tls-certs\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.125005 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.124873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-dshm\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.125005 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.124891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-model-cache\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.125005 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.124934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-tmp-dir\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.125403 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.125380    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-home\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.125510 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.125428    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-model-cache\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.125510 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.125437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-tmp-dir\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.127105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.127087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-dshm\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.127368 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.127352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12dd8566-525c-4af9-84a6-418479e879aa-tls-certs\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.132150 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.132132    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht69d\" (UniqueName: \"kubernetes.io/projected/12dd8566-525c-4af9-84a6-418479e879aa-kube-api-access-ht69d\") pod \"ipp-route-f0171b-kserve-6859cdbfd7-shb68\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") " pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.254054 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.253981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:47.524647 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:47.524621    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"]
Sep 15 12:58:47.526494 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:58:47.526466    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12dd8566_525c_4af9_84a6_418479e879aa.slice/crio-3a3bb37e0eaafab2fa7d0ee24bfc362ecc55b383f81cd828f01c01e66cd4204c WatchSource:0}: Error finding container 3a3bb37e0eaafab2fa7d0ee24bfc362ecc55b383f81cd828f01c01e66cd4204c: Status 404 returned error can't find the container with id 3a3bb37e0eaafab2fa7d0ee24bfc362ecc55b383f81cd828f01c01e66cd4204c
Sep 15 12:58:48.536998 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:48.536965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68" event={"ID":"12dd8566-525c-4af9-84a6-418479e879aa","Type":"ContainerStarted","Data":"c4d1facdc9b5c9afd6c7eec3db5cc7f5ed1fef71c842402423783d767159e90a"}
Sep 15 12:58:48.537398 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:48.537006    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68" event={"ID":"12dd8566-525c-4af9-84a6-418479e879aa","Type":"ContainerStarted","Data":"3a3bb37e0eaafab2fa7d0ee24bfc362ecc55b383f81cd828f01c01e66cd4204c"}
Sep 15 12:58:48.553989 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:48.553938    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68" podStartSLOduration=2.553921645 podStartE2EDuration="2.553921645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:58:48.551701653 +0000 UTC m=+1999.936412539" watchObservedRunningTime="2026-09-15 12:58:48.553921645 +0000 UTC m=+1999.938632569"
Sep 15 12:58:51.055388 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:51.055356    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:58:57.254928 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:57.254896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:57.255515 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:57.254971    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:57.267556 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:57.267535    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:58:57.594013 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:58:57.593940    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 12:59:17.547860 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:17.547809    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:59:24.201314 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:24.201280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:24.584043 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:24.583954    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:24.814639 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:24.814595    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:26.882950 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:26.882911    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:27.506700 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:27.506661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:27.603128 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:27.603094    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:27.698105 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:27.698070    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:28.379634 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:28.379597    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:29.011815 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:29.011780    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:29.791599 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:29.791565    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:30.454245 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.454209    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"]
Sep 15 12:59:30.458483 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.458463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:30.460896 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.460870    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-85cea9-serving-cert\""
Sep 15 12:59:30.467206 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.466260    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"]
Sep 15 12:59:30.514127 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.514092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/286bcdce-f019-4b85-950d-00df2cf8ae83-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:30.514305 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.514222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/286bcdce-f019-4b85-950d-00df2cf8ae83-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:30.514305 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.514296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286bcdce-f019-4b85-950d-00df2cf8ae83-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:30.514433 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.514361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmk5t\" (UniqueName: \"kubernetes.io/projected/286bcdce-f019-4b85-950d-00df2cf8ae83-kube-api-access-dmk5t\") pod \"maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:30.578443 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.578403    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:30.615158 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.615126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/286bcdce-f019-4b85-950d-00df2cf8ae83-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:30.615337 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.615167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286bcdce-f019-4b85-950d-00df2cf8ae83-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:30.615337 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.615193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmk5t\" (UniqueName: \"kubernetes.io/projected/286bcdce-f019-4b85-950d-00df2cf8ae83-kube-api-access-dmk5t\") pod \"maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:30.615337 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.615216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/286bcdce-f019-4b85-950d-00df2cf8ae83-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:30.615337 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:59:30.615303    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-serving-cert: secret "maas-api-e2e-ipp-cleanup-85cea9-serving-cert" not found
Sep 15 12:59:30.615587 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:59:30.615357    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/286bcdce-f019-4b85-950d-00df2cf8ae83-maas-api-tls podName:286bcdce-f019-4b85-950d-00df2cf8ae83 nodeName:}" failed. No retries permitted until 2026-09-15 12:59:31.115340923 +0000 UTC m=+2042.500051785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/286bcdce-f019-4b85-950d-00df2cf8ae83-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk" (UID: "286bcdce-f019-4b85-950d-00df2cf8ae83") : secret "maas-api-e2e-ipp-cleanup-85cea9-serving-cert" not found
Sep 15 12:59:30.615922 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.615900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/286bcdce-f019-4b85-950d-00df2cf8ae83-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:30.617707 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.617685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286bcdce-f019-4b85-950d-00df2cf8ae83-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:30.633048 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.633028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmk5t\" (UniqueName: \"kubernetes.io/projected/286bcdce-f019-4b85-950d-00df2cf8ae83-kube-api-access-dmk5t\") pod \"maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:30.720402 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.720308    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:30.820291 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:30.820247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:31.050813 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:31.050726    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:31.119760 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:31.119727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/286bcdce-f019-4b85-950d-00df2cf8ae83-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:31.122358 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:31.122329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/286bcdce-f019-4b85-950d-00df2cf8ae83-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:31.153083 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:31.153050    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:31.371793 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:31.371753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:31.650987 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:31.650962    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"]
Sep 15 12:59:31.652910 ip-10-0-129-17 kubenswrapper[2579]: W0915 12:59:31.652882    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286bcdce_f019_4b85_950d_00df2cf8ae83.slice/crio-eb1b09c1d0f529823ddb76b5537700f5566b646e67e2d488d4490ad82989bec2 WatchSource:0}: Error finding container eb1b09c1d0f529823ddb76b5537700f5566b646e67e2d488d4490ad82989bec2: Status 404 returned error can't find the container with id eb1b09c1d0f529823ddb76b5537700f5566b646e67e2d488d4490ad82989bec2
Sep 15 12:59:31.757643 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:31.757607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk" event={"ID":"286bcdce-f019-4b85-950d-00df2cf8ae83","Type":"ContainerStarted","Data":"eb1b09c1d0f529823ddb76b5537700f5566b646e67e2d488d4490ad82989bec2"}
Sep 15 12:59:32.764812 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:32.764776    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk" event={"ID":"286bcdce-f019-4b85-950d-00df2cf8ae83","Type":"ContainerStarted","Data":"5d5a44ae5961ad200444ed5ec01a3fecc259c82796a210aa812684cea39f2712"}
Sep 15 12:59:32.765296 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:32.764872    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:32.787088 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:32.787034    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk" podStartSLOduration=2.335114581 podStartE2EDuration="2.787020547s" totalImagesPullingTime="451.905966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:59:32.785142557 +0000 UTC m=+2044.169853465" watchObservedRunningTime="2026-09-15 12:59:32.787020547 +0000 UTC m=+2044.171731431"
Sep 15 12:59:38.260853 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:38.259876    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 12:59:38.775856 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:38.775801    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:52.489947 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:52.489870    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:52.605303 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:52.605267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:53.254663 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:53.254623    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:53.991794 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:53.991755    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:54.794322 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:54.794282    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:55.445539 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:55.445495    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:56.088061 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:56.088024    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:56.847107 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:56.847074    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:57.563431 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:57.563394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:58.832311 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:58.832280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:59.353465 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.353427    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"]
Sep 15 12:59:59.353735 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.353685    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk" podUID="286bcdce-f019-4b85-950d-00df2cf8ae83" containerName="maas-api" containerID="cri-o://5d5a44ae5961ad200444ed5ec01a3fecc259c82796a210aa812684cea39f2712" gracePeriod=30
Sep 15 12:59:59.371207 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.371177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:59.471194 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.470913    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:59.566531 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.566496    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:59.600264 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.600243    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:59.627689 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.627617    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:59.673402 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.673373    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/286bcdce-f019-4b85-950d-00df2cf8ae83-openshift-service-ca\") pod \"286bcdce-f019-4b85-950d-00df2cf8ae83\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") "
Sep 15 12:59:59.673533 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.673418    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/286bcdce-f019-4b85-950d-00df2cf8ae83-maas-api-tls\") pod \"286bcdce-f019-4b85-950d-00df2cf8ae83\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") "
Sep 15 12:59:59.673533 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.673488    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286bcdce-f019-4b85-950d-00df2cf8ae83-metrics-certs\") pod \"286bcdce-f019-4b85-950d-00df2cf8ae83\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") "
Sep 15 12:59:59.673533 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.673516    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmk5t\" (UniqueName: \"kubernetes.io/projected/286bcdce-f019-4b85-950d-00df2cf8ae83-kube-api-access-dmk5t\") pod \"286bcdce-f019-4b85-950d-00df2cf8ae83\" (UID: \"286bcdce-f019-4b85-950d-00df2cf8ae83\") "
Sep 15 12:59:59.673803 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.673776    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/286bcdce-f019-4b85-950d-00df2cf8ae83-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "286bcdce-f019-4b85-950d-00df2cf8ae83" (UID: "286bcdce-f019-4b85-950d-00df2cf8ae83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:59:59.675622 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.675598    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/286bcdce-f019-4b85-950d-00df2cf8ae83-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "286bcdce-f019-4b85-950d-00df2cf8ae83" (UID: "286bcdce-f019-4b85-950d-00df2cf8ae83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:59:59.675689 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.675617    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286bcdce-f019-4b85-950d-00df2cf8ae83-kube-api-access-dmk5t" (OuterVolumeSpecName: "kube-api-access-dmk5t") pod "286bcdce-f019-4b85-950d-00df2cf8ae83" (UID: "286bcdce-f019-4b85-950d-00df2cf8ae83"). InnerVolumeSpecName "kube-api-access-dmk5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:59:59.675689 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.675680    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/286bcdce-f019-4b85-950d-00df2cf8ae83-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "286bcdce-f019-4b85-950d-00df2cf8ae83" (UID: "286bcdce-f019-4b85-950d-00df2cf8ae83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:59:59.774892 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.774863    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/286bcdce-f019-4b85-950d-00df2cf8ae83-maas-api-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:59:59.774892 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.774890    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/286bcdce-f019-4b85-950d-00df2cf8ae83-metrics-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:59:59.774892 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.774900    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmk5t\" (UniqueName: \"kubernetes.io/projected/286bcdce-f019-4b85-950d-00df2cf8ae83-kube-api-access-dmk5t\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:59:59.775118 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.774909    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/286bcdce-f019-4b85-950d-00df2cf8ae83-openshift-service-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 12:59:59.904679 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.904596    2579 generic.go:356] "Generic (PLEG): container finished" podID="286bcdce-f019-4b85-950d-00df2cf8ae83" containerID="5d5a44ae5961ad200444ed5ec01a3fecc259c82796a210aa812684cea39f2712" exitCode=0
Sep 15 12:59:59.904679 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.904647    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"
Sep 15 12:59:59.905192 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.904697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk" event={"ID":"286bcdce-f019-4b85-950d-00df2cf8ae83","Type":"ContainerDied","Data":"5d5a44ae5961ad200444ed5ec01a3fecc259c82796a210aa812684cea39f2712"}
Sep 15 12:59:59.905192 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.904755    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk" event={"ID":"286bcdce-f019-4b85-950d-00df2cf8ae83","Type":"ContainerDied","Data":"eb1b09c1d0f529823ddb76b5537700f5566b646e67e2d488d4490ad82989bec2"}
Sep 15 12:59:59.905192 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.904789    2579 scope.go:117] "RemoveContainer" containerID="5d5a44ae5961ad200444ed5ec01a3fecc259c82796a210aa812684cea39f2712"
Sep 15 12:59:59.914514 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.914499    2579 scope.go:117] "RemoveContainer" containerID="5d5a44ae5961ad200444ed5ec01a3fecc259c82796a210aa812684cea39f2712"
Sep 15 12:59:59.914755 ip-10-0-129-17 kubenswrapper[2579]: E0915 12:59:59.914739    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5a44ae5961ad200444ed5ec01a3fecc259c82796a210aa812684cea39f2712\": container with ID starting with 5d5a44ae5961ad200444ed5ec01a3fecc259c82796a210aa812684cea39f2712 not found: ID does not exist" containerID="5d5a44ae5961ad200444ed5ec01a3fecc259c82796a210aa812684cea39f2712"
Sep 15 12:59:59.914804 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.914762    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5a44ae5961ad200444ed5ec01a3fecc259c82796a210aa812684cea39f2712"} err="failed to get container status \"5d5a44ae5961ad200444ed5ec01a3fecc259c82796a210aa812684cea39f2712\": rpc error: code = NotFound desc = could not find container \"5d5a44ae5961ad200444ed5ec01a3fecc259c82796a210aa812684cea39f2712\": container with ID starting with 5d5a44ae5961ad200444ed5ec01a3fecc259c82796a210aa812684cea39f2712 not found: ID does not exist"
Sep 15 12:59:59.918107 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.918084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 12:59:59.925847 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.925807    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"]
Sep 15 12:59:59.929558 ip-10-0-129-17 kubenswrapper[2579]: I0915 12:59:59.929538    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-85cea9-b7c84f66-4wblk"]
Sep 15 13:00:00.018450 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.018410    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:00.133544 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.133510    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824620-8t6hx"]
Sep 15 13:00:00.134142 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.134118    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="286bcdce-f019-4b85-950d-00df2cf8ae83" containerName="maas-api"
Sep 15 13:00:00.134323 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.134307    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="286bcdce-f019-4b85-950d-00df2cf8ae83" containerName="maas-api"
Sep 15 13:00:00.134547 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.134522    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="286bcdce-f019-4b85-950d-00df2cf8ae83" containerName="maas-api"
Sep 15 13:00:00.138899 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.138877    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-17f8f8-29824620-bhdhm"]
Sep 15 13:00:00.138998 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.138980    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824620-8t6hx"
Sep 15 13:00:00.141324 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.141304    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\""
Sep 15 13:00:00.142714 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.142682    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-17f8f8-29824620-bhdhm"
Sep 15 13:00:00.145870 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.145846    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824620-8t6hx"]
Sep 15 13:00:00.152228 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.152202    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824620-8t6hx"
Sep 15 13:00:00.160064 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.160000    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-17f8f8-29824620-bhdhm"
Sep 15 13:00:00.160343 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.160320    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-17f8f8-29824620-bhdhm"]
Sep 15 13:00:00.425439 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.425407    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824620-8t6hx"]
Sep 15 13:00:00.427596 ip-10-0-129-17 kubenswrapper[2579]: W0915 13:00:00.427572    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80481b6_a19b_447b_b6d2_2410d159369a.slice/crio-89325d57edd2fa979ed8b35616ba1e6c920609e027f6a8065bcf59c1e0b2f405 WatchSource:0}: Error finding container 89325d57edd2fa979ed8b35616ba1e6c920609e027f6a8065bcf59c1e0b2f405: Status 404 returned error can't find the container with id 89325d57edd2fa979ed8b35616ba1e6c920609e027f6a8065bcf59c1e0b2f405
Sep 15 13:00:00.437620 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.437601    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-17f8f8-29824620-bhdhm"]
Sep 15 13:00:00.441075 ip-10-0-129-17 kubenswrapper[2579]: W0915 13:00:00.441048    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5ef640_3f07_4bab_9a20_bbf6423386ce.slice/crio-7e1a9119d1f9e32261cb4a500fb6319607da62d745a8a1d3bea076707812e4b3 WatchSource:0}: Error finding container 7e1a9119d1f9e32261cb4a500fb6319607da62d745a8a1d3bea076707812e4b3: Status 404 returned error can't find the container with id 7e1a9119d1f9e32261cb4a500fb6319607da62d745a8a1d3bea076707812e4b3
Sep 15 13:00:00.776960 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.776932    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:00.918617 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.918583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824620-8t6hx" event={"ID":"b80481b6-a19b-447b-b6d2-2410d159369a","Type":"ContainerStarted","Data":"344e6df3643fdd97d91137313cd23bcacf9ff11ff17da9ac9586b1b78b95a381"}
Sep 15 13:00:00.919161 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.918665    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824620-8t6hx" event={"ID":"b80481b6-a19b-447b-b6d2-2410d159369a","Type":"ContainerStarted","Data":"89325d57edd2fa979ed8b35616ba1e6c920609e027f6a8065bcf59c1e0b2f405"}
Sep 15 13:00:00.920210 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.920178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-17f8f8-29824620-bhdhm" event={"ID":"4a5ef640-3f07-4bab-9a20-bbf6423386ce","Type":"ContainerStarted","Data":"478373bb3e41fe1bb58f0d941c0b94c9481625d0b9a2046db3ce616f20d8a340"}
Sep 15 13:00:00.920337 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.920231    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-17f8f8-29824620-bhdhm" event={"ID":"4a5ef640-3f07-4bab-9a20-bbf6423386ce","Type":"ContainerStarted","Data":"7e1a9119d1f9e32261cb4a500fb6319607da62d745a8a1d3bea076707812e4b3"}
Sep 15 13:00:00.936467 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.936421    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824620-8t6hx" podStartSLOduration=0.93640765 podStartE2EDuration="936.40765ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:00:00.933230359 +0000 UTC m=+2072.317941244" watchObservedRunningTime="2026-09-15 13:00:00.93640765 +0000 UTC m=+2072.321118534"
Sep 15 13:00:00.949671 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:00.949626    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-17f8f8-29824620-bhdhm" podStartSLOduration=0.949614712 podStartE2EDuration="949.614712ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:00:00.948633194 +0000 UTC m=+2072.333344080" watchObservedRunningTime="2026-09-15 13:00:00.949614712 +0000 UTC m=+2072.334325645"
Sep 15 13:00:01.106084 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:01.106055    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="286bcdce-f019-4b85-950d-00df2cf8ae83" path="/var/lib/kubelet/pods/286bcdce-f019-4b85-950d-00df2cf8ae83/volumes"
Sep 15 13:00:01.620348 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:01.620244    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:01.928093 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:01.928004    2579 generic.go:356] "Generic (PLEG): container finished" podID="4a5ef640-3f07-4bab-9a20-bbf6423386ce" containerID="478373bb3e41fe1bb58f0d941c0b94c9481625d0b9a2046db3ce616f20d8a340" exitCode=0
Sep 15 13:00:01.928527 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:01.928089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-17f8f8-29824620-bhdhm" event={"ID":"4a5ef640-3f07-4bab-9a20-bbf6423386ce","Type":"ContainerDied","Data":"478373bb3e41fe1bb58f0d941c0b94c9481625d0b9a2046db3ce616f20d8a340"}
Sep 15 13:00:01.929634 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:01.929616    2579 generic.go:356] "Generic (PLEG): container finished" podID="b80481b6-a19b-447b-b6d2-2410d159369a" containerID="344e6df3643fdd97d91137313cd23bcacf9ff11ff17da9ac9586b1b78b95a381" exitCode=0
Sep 15 13:00:01.929707 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:01.929660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824620-8t6hx" event={"ID":"b80481b6-a19b-447b-b6d2-2410d159369a","Type":"ContainerDied","Data":"344e6df3643fdd97d91137313cd23bcacf9ff11ff17da9ac9586b1b78b95a381"}
Sep 15 13:00:02.546199 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:02.546164    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:03.070109 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:03.070084    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824620-8t6hx"
Sep 15 13:00:03.104167 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:03.104141    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-17f8f8-29824620-bhdhm"
Sep 15 13:00:03.681275 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:03.681239    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:03.942161 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:03.942076    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824620-8t6hx" event={"ID":"b80481b6-a19b-447b-b6d2-2410d159369a","Type":"ContainerDied","Data":"89325d57edd2fa979ed8b35616ba1e6c920609e027f6a8065bcf59c1e0b2f405"}
Sep 15 13:00:03.942161 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:03.942096    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824620-8t6hx"
Sep 15 13:00:03.942161 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:03.942126    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89325d57edd2fa979ed8b35616ba1e6c920609e027f6a8065bcf59c1e0b2f405"
Sep 15 13:00:03.943615 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:03.943597    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-17f8f8-29824620-bhdhm"
Sep 15 13:00:03.943717 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:03.943630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-17f8f8-29824620-bhdhm" event={"ID":"4a5ef640-3f07-4bab-9a20-bbf6423386ce","Type":"ContainerDied","Data":"7e1a9119d1f9e32261cb4a500fb6319607da62d745a8a1d3bea076707812e4b3"}
Sep 15 13:00:03.943717 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:03.943675    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e1a9119d1f9e32261cb4a500fb6319607da62d745a8a1d3bea076707812e4b3"
Sep 15 13:00:03.955674 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:03.955652    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:00:05.925064 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:05.925028    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:05.992202 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:05.992168    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:10.173517 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:10.173473    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:10.979407 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:10.979376    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:11.757683 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:11.757641    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:12.559618 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:12.559580    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:15.178640 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:15.178608    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:17.343664 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:17.343633    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:22.853066 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:22.853027    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:00:24.948117 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:24.948073    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:25.044805 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:25.044769    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:25.113641 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:25.113602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:25.164904 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:25.164870    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:25.472151 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:25.472120    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:25.572510 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:25.572474    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:27.293991 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:27.293956    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:27.392905 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:27.392869    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:27.700436 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:27.700402    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:29.298608 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:29.298582    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 13:00:29.300963 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:29.300944    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 13:00:30.456749 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:30.456710    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"]
Sep 15 13:00:30.500937 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:30.500901    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:00:34.953431 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:34.953394    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:00:45.476053 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.476007    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68" podUID="12dd8566-525c-4af9-84a6-418479e879aa" containerName="main" containerID="cri-o://c4d1facdc9b5c9afd6c7eec3db5cc7f5ed1fef71c842402423783d767159e90a" gracePeriod=45
Sep 15 13:00:45.727591 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.727533    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 13:00:45.794601 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.794565    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-home\") pod \"12dd8566-525c-4af9-84a6-418479e879aa\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") "
Sep 15 13:00:45.794773 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.794641    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12dd8566-525c-4af9-84a6-418479e879aa-tls-certs\") pod \"12dd8566-525c-4af9-84a6-418479e879aa\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") "
Sep 15 13:00:45.794773 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.794673    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-dshm\") pod \"12dd8566-525c-4af9-84a6-418479e879aa\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") "
Sep 15 13:00:45.794773 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.794702    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-model-cache\") pod \"12dd8566-525c-4af9-84a6-418479e879aa\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") "
Sep 15 13:00:45.794773 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.794736    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht69d\" (UniqueName: \"kubernetes.io/projected/12dd8566-525c-4af9-84a6-418479e879aa-kube-api-access-ht69d\") pod \"12dd8566-525c-4af9-84a6-418479e879aa\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") "
Sep 15 13:00:45.795025 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.794847    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-tmp-dir\") pod \"12dd8566-525c-4af9-84a6-418479e879aa\" (UID: \"12dd8566-525c-4af9-84a6-418479e879aa\") "
Sep 15 13:00:45.795175 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.795138    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-home" (OuterVolumeSpecName: "home") pod "12dd8566-525c-4af9-84a6-418479e879aa" (UID: "12dd8566-525c-4af9-84a6-418479e879aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:00:45.795175 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.795167    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "12dd8566-525c-4af9-84a6-418479e879aa" (UID: "12dd8566-525c-4af9-84a6-418479e879aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:00:45.795373 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.795181    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12dd8566-525c-4af9-84a6-418479e879aa" (UID: "12dd8566-525c-4af9-84a6-418479e879aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:00:45.796936 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.796912    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12dd8566-525c-4af9-84a6-418479e879aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12dd8566-525c-4af9-84a6-418479e879aa" (UID: "12dd8566-525c-4af9-84a6-418479e879aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:00:45.797051 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.796994    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-dshm" (OuterVolumeSpecName: "dshm") pod "12dd8566-525c-4af9-84a6-418479e879aa" (UID: "12dd8566-525c-4af9-84a6-418479e879aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:00:45.797051 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.797041    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12dd8566-525c-4af9-84a6-418479e879aa-kube-api-access-ht69d" (OuterVolumeSpecName: "kube-api-access-ht69d") pod "12dd8566-525c-4af9-84a6-418479e879aa" (UID: "12dd8566-525c-4af9-84a6-418479e879aa"). InnerVolumeSpecName "kube-api-access-ht69d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:00:45.896329 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.896293    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:00:45.896329 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.896328    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:00:45.896557 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.896339    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12dd8566-525c-4af9-84a6-418479e879aa-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:00:45.896557 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.896352    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:00:45.896557 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.896363    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12dd8566-525c-4af9-84a6-418479e879aa-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:00:45.896557 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:45.896376    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht69d\" (UniqueName: \"kubernetes.io/projected/12dd8566-525c-4af9-84a6-418479e879aa-kube-api-access-ht69d\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:00:46.175520 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:46.175486    2579 generic.go:356] "Generic (PLEG): container finished" podID="12dd8566-525c-4af9-84a6-418479e879aa" containerID="c4d1facdc9b5c9afd6c7eec3db5cc7f5ed1fef71c842402423783d767159e90a" exitCode=0
Sep 15 13:00:46.175695 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:46.175526    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"
Sep 15 13:00:46.175695 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:46.175537    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68" event={"ID":"12dd8566-525c-4af9-84a6-418479e879aa","Type":"ContainerDied","Data":"c4d1facdc9b5c9afd6c7eec3db5cc7f5ed1fef71c842402423783d767159e90a"}
Sep 15 13:00:46.175695 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:46.175576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68" event={"ID":"12dd8566-525c-4af9-84a6-418479e879aa","Type":"ContainerDied","Data":"3a3bb37e0eaafab2fa7d0ee24bfc362ecc55b383f81cd828f01c01e66cd4204c"}
Sep 15 13:00:46.175695 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:46.175594    2579 scope.go:117] "RemoveContainer" containerID="c4d1facdc9b5c9afd6c7eec3db5cc7f5ed1fef71c842402423783d767159e90a"
Sep 15 13:00:46.184888 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:46.184873    2579 scope.go:117] "RemoveContainer" containerID="c4d1facdc9b5c9afd6c7eec3db5cc7f5ed1fef71c842402423783d767159e90a"
Sep 15 13:00:46.185165 ip-10-0-129-17 kubenswrapper[2579]: E0915 13:00:46.185147    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d1facdc9b5c9afd6c7eec3db5cc7f5ed1fef71c842402423783d767159e90a\": container with ID starting with c4d1facdc9b5c9afd6c7eec3db5cc7f5ed1fef71c842402423783d767159e90a not found: ID does not exist" containerID="c4d1facdc9b5c9afd6c7eec3db5cc7f5ed1fef71c842402423783d767159e90a"
Sep 15 13:00:46.185215 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:46.185174    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d1facdc9b5c9afd6c7eec3db5cc7f5ed1fef71c842402423783d767159e90a"} err="failed to get container status \"c4d1facdc9b5c9afd6c7eec3db5cc7f5ed1fef71c842402423783d767159e90a\": rpc error: code = NotFound desc = could not find container \"c4d1facdc9b5c9afd6c7eec3db5cc7f5ed1fef71c842402423783d767159e90a\": container with ID starting with c4d1facdc9b5c9afd6c7eec3db5cc7f5ed1fef71c842402423783d767159e90a not found: ID does not exist"
Sep 15 13:00:46.197299 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:46.197274    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"]
Sep 15 13:00:46.199490 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:46.199469    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f0171b/ipp-route-f0171b-kserve-6859cdbfd7-shb68"]
Sep 15 13:00:47.105584 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:00:47.105556    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12dd8566-525c-4af9-84a6-418479e879aa" path="/var/lib/kubelet/pods/12dd8566-525c-4af9-84a6-418479e879aa/volumes"
Sep 15 13:01:06.980036 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:06.979992    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:14.232004 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:14.231960    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:14.396417 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:14.396377    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:17.375409 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:17.375370    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:17.427684 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:17.427650    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:19.557524 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:19.557488    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:19.657989 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:19.657958    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:24.896959 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:24.896872    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"]
Sep 15 13:01:24.897425 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:24.897136    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85" podUID="79efd4dc-69e1-4262-b7d5-20914de57933" containerName="maas-api" containerID="cri-o://8a9319756760591e4e01d1dee1fef99c34674d7bb97392e86ad41e5146756b6e" gracePeriod=30
Sep 15 13:01:24.927610 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:24.927576    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:25.141417 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.141392    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 13:01:25.243879 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.243773    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79efd4dc-69e1-4262-b7d5-20914de57933-maas-api-tls\") pod \"79efd4dc-69e1-4262-b7d5-20914de57933\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") "
Sep 15 13:01:25.243879 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.243853    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79efd4dc-69e1-4262-b7d5-20914de57933-metrics-certs\") pod \"79efd4dc-69e1-4262-b7d5-20914de57933\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") "
Sep 15 13:01:25.244136 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.243897    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79efd4dc-69e1-4262-b7d5-20914de57933-openshift-service-ca\") pod \"79efd4dc-69e1-4262-b7d5-20914de57933\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") "
Sep 15 13:01:25.244136 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.243963    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncvww\" (UniqueName: \"kubernetes.io/projected/79efd4dc-69e1-4262-b7d5-20914de57933-kube-api-access-ncvww\") pod \"79efd4dc-69e1-4262-b7d5-20914de57933\" (UID: \"79efd4dc-69e1-4262-b7d5-20914de57933\") "
Sep 15 13:01:25.244298 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.244268    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79efd4dc-69e1-4262-b7d5-20914de57933-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79efd4dc-69e1-4262-b7d5-20914de57933" (UID: "79efd4dc-69e1-4262-b7d5-20914de57933"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 13:01:25.245983 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.245953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79efd4dc-69e1-4262-b7d5-20914de57933-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79efd4dc-69e1-4262-b7d5-20914de57933" (UID: "79efd4dc-69e1-4262-b7d5-20914de57933"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:01:25.246093 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.245988    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79efd4dc-69e1-4262-b7d5-20914de57933-kube-api-access-ncvww" (OuterVolumeSpecName: "kube-api-access-ncvww") pod "79efd4dc-69e1-4262-b7d5-20914de57933" (UID: "79efd4dc-69e1-4262-b7d5-20914de57933"). InnerVolumeSpecName "kube-api-access-ncvww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:01:25.246093 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.246020    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79efd4dc-69e1-4262-b7d5-20914de57933-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "79efd4dc-69e1-4262-b7d5-20914de57933" (UID: "79efd4dc-69e1-4262-b7d5-20914de57933"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:01:25.345526 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.345498    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79efd4dc-69e1-4262-b7d5-20914de57933-maas-api-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:01:25.345526 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.345521    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79efd4dc-69e1-4262-b7d5-20914de57933-metrics-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:01:25.345526 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.345530    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79efd4dc-69e1-4262-b7d5-20914de57933-openshift-service-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:01:25.345743 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.345539    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncvww\" (UniqueName: \"kubernetes.io/projected/79efd4dc-69e1-4262-b7d5-20914de57933-kube-api-access-ncvww\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:01:25.380859 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.380809    2579 generic.go:356] "Generic (PLEG): container finished" podID="79efd4dc-69e1-4262-b7d5-20914de57933" containerID="8a9319756760591e4e01d1dee1fef99c34674d7bb97392e86ad41e5146756b6e" exitCode=0
Sep 15 13:01:25.381009 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.380892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85" event={"ID":"79efd4dc-69e1-4262-b7d5-20914de57933","Type":"ContainerDied","Data":"8a9319756760591e4e01d1dee1fef99c34674d7bb97392e86ad41e5146756b6e"}
Sep 15 13:01:25.381009 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.380909    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"
Sep 15 13:01:25.381009 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.380953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85" event={"ID":"79efd4dc-69e1-4262-b7d5-20914de57933","Type":"ContainerDied","Data":"c4df3f5a5eb110abff9ce01d3dd7c83e7697dc0dce7c5f15829f3f3f53473f90"}
Sep 15 13:01:25.381009 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.380978    2579 scope.go:117] "RemoveContainer" containerID="8a9319756760591e4e01d1dee1fef99c34674d7bb97392e86ad41e5146756b6e"
Sep 15 13:01:25.390537 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.390507    2579 scope.go:117] "RemoveContainer" containerID="8a9319756760591e4e01d1dee1fef99c34674d7bb97392e86ad41e5146756b6e"
Sep 15 13:01:25.390818 ip-10-0-129-17 kubenswrapper[2579]: E0915 13:01:25.390796    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9319756760591e4e01d1dee1fef99c34674d7bb97392e86ad41e5146756b6e\": container with ID starting with 8a9319756760591e4e01d1dee1fef99c34674d7bb97392e86ad41e5146756b6e not found: ID does not exist" containerID="8a9319756760591e4e01d1dee1fef99c34674d7bb97392e86ad41e5146756b6e"
Sep 15 13:01:25.390910 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.390827    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9319756760591e4e01d1dee1fef99c34674d7bb97392e86ad41e5146756b6e"} err="failed to get container status \"8a9319756760591e4e01d1dee1fef99c34674d7bb97392e86ad41e5146756b6e\": rpc error: code = NotFound desc = could not find container \"8a9319756760591e4e01d1dee1fef99c34674d7bb97392e86ad41e5146756b6e\": container with ID starting with 8a9319756760591e4e01d1dee1fef99c34674d7bb97392e86ad41e5146756b6e not found: ID does not exist"
Sep 15 13:01:25.402553 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.402526    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"]
Sep 15 13:01:25.406391 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:25.406368    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cd72a9-d7474695-msj85"]
Sep 15 13:01:26.874489 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:26.874451    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:26.975604 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:26.975562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:27.105199 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:27.105169    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79efd4dc-69e1-4262-b7d5-20914de57933" path="/var/lib/kubelet/pods/79efd4dc-69e1-4262-b7d5-20914de57933/volumes"
Sep 15 13:01:27.169872 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:27.169762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:28.450594 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:28.450553    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:01:31.483928 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:31.483887    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:31.588215 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:31.588182    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:31.761770 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:31.761684    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:32.781711 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:32.781675    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:33.582739 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:33.582706    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:34.236994 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:34.236961    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:42.607343 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:42.607300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:42.678078 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:42.678033    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:42.806725 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:42.806693    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:42.866583 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:42.866495    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:43.175106 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:43.175017    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:43.275328 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:43.275292    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:43.975839 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:43.975797    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:45.549588 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:45.549553    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:01:49.010382 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:49.010347    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:51.290005 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:51.289968    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:01:51.391175 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:01:51.391140    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:02:00.100715 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.100685    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"]
Sep 15 13:02:00.101206 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.100940    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75" podUID="136a2d5c-2cb7-4f35-8df6-d1eb52af7467" containerName="maas-api" containerID="cri-o://c07b0dd6c08834da1318dd5d7232e88fced2cf56b80b4ee9e43428240a6b0cd8" gracePeriod=30
Sep 15 13:02:00.120742 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.120713    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-17f8f8-29824620-bhdhm"]
Sep 15 13:02:00.125808 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.125785    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-17f8f8-29824620-bhdhm"]
Sep 15 13:02:00.125938 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.125854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:02:00.188516 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.188485    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:02:00.288140 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.288105    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:02:00.342301 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.342280    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 13:02:00.363983 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.363866    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:02:00.459176 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.459143    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-openshift-service-ca\") pod \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") "
Sep 15 13:02:00.459363 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.459191    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8dth\" (UniqueName: \"kubernetes.io/projected/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-kube-api-access-k8dth\") pod \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") "
Sep 15 13:02:00.459363 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.459252    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-maas-api-tls\") pod \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") "
Sep 15 13:02:00.459363 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.459295    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-metrics-certs\") pod \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\" (UID: \"136a2d5c-2cb7-4f35-8df6-d1eb52af7467\") "
Sep 15 13:02:00.459556 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.459529    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "136a2d5c-2cb7-4f35-8df6-d1eb52af7467" (UID: "136a2d5c-2cb7-4f35-8df6-d1eb52af7467"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 13:02:00.461343 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.461319    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "136a2d5c-2cb7-4f35-8df6-d1eb52af7467" (UID: "136a2d5c-2cb7-4f35-8df6-d1eb52af7467"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:02:00.461456 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.461376    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-kube-api-access-k8dth" (OuterVolumeSpecName: "kube-api-access-k8dth") pod "136a2d5c-2cb7-4f35-8df6-d1eb52af7467" (UID: "136a2d5c-2cb7-4f35-8df6-d1eb52af7467"). InnerVolumeSpecName "kube-api-access-k8dth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:02:00.461456 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.461396    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "136a2d5c-2cb7-4f35-8df6-d1eb52af7467" (UID: "136a2d5c-2cb7-4f35-8df6-d1eb52af7467"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:02:00.560187 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.560151    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8dth\" (UniqueName: \"kubernetes.io/projected/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-kube-api-access-k8dth\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:02:00.560187 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.560184    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-maas-api-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:02:00.560390 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.560198    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-metrics-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:02:00.560390 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.560211    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/136a2d5c-2cb7-4f35-8df6-d1eb52af7467-openshift-service-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:02:00.562554 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.562526    2579 generic.go:356] "Generic (PLEG): container finished" podID="136a2d5c-2cb7-4f35-8df6-d1eb52af7467" containerID="c07b0dd6c08834da1318dd5d7232e88fced2cf56b80b4ee9e43428240a6b0cd8" exitCode=0
Sep 15 13:02:00.562655 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.562580    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"
Sep 15 13:02:00.562655 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.562581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75" event={"ID":"136a2d5c-2cb7-4f35-8df6-d1eb52af7467","Type":"ContainerDied","Data":"c07b0dd6c08834da1318dd5d7232e88fced2cf56b80b4ee9e43428240a6b0cd8"}
Sep 15 13:02:00.562655 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.562635    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75" event={"ID":"136a2d5c-2cb7-4f35-8df6-d1eb52af7467","Type":"ContainerDied","Data":"36f72af98f244b04cbd04d0e652418998f1512f285f551276e7bff4ab638f40f"}
Sep 15 13:02:00.562755 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.562664    2579 scope.go:117] "RemoveContainer" containerID="c07b0dd6c08834da1318dd5d7232e88fced2cf56b80b4ee9e43428240a6b0cd8"
Sep 15 13:02:00.571971 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.571952    2579 scope.go:117] "RemoveContainer" containerID="c07b0dd6c08834da1318dd5d7232e88fced2cf56b80b4ee9e43428240a6b0cd8"
Sep 15 13:02:00.572224 ip-10-0-129-17 kubenswrapper[2579]: E0915 13:02:00.572205    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07b0dd6c08834da1318dd5d7232e88fced2cf56b80b4ee9e43428240a6b0cd8\": container with ID starting with c07b0dd6c08834da1318dd5d7232e88fced2cf56b80b4ee9e43428240a6b0cd8 not found: ID does not exist" containerID="c07b0dd6c08834da1318dd5d7232e88fced2cf56b80b4ee9e43428240a6b0cd8"
Sep 15 13:02:00.572270 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.572230    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07b0dd6c08834da1318dd5d7232e88fced2cf56b80b4ee9e43428240a6b0cd8"} err="failed to get container status \"c07b0dd6c08834da1318dd5d7232e88fced2cf56b80b4ee9e43428240a6b0cd8\": rpc error: code = NotFound desc = could not find container \"c07b0dd6c08834da1318dd5d7232e88fced2cf56b80b4ee9e43428240a6b0cd8\": container with ID starting with c07b0dd6c08834da1318dd5d7232e88fced2cf56b80b4ee9e43428240a6b0cd8 not found: ID does not exist"
Sep 15 13:02:00.586742 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.586719    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"]
Sep 15 13:02:00.598045 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.598022    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17f8f8-7fdfdf6dd4-9qp75"]
Sep 15 13:02:00.695746 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.695658    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:02:00.797914 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:00.797882    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:02:01.105262 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:01.105226    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="136a2d5c-2cb7-4f35-8df6-d1eb52af7467" path="/var/lib/kubelet/pods/136a2d5c-2cb7-4f35-8df6-d1eb52af7467/volumes"
Sep 15 13:02:01.105656 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:01.105522    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a5ef640-3f07-4bab-9a20-bbf6423386ce" path="/var/lib/kubelet/pods/4a5ef640-3f07-4bab-9a20-bbf6423386ce/volumes"
Sep 15 13:02:02.756771 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:02.756737    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:02:06.509842 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:06.509791    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:02:06.572928 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:06.572896    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:02:09.561600 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:09.561565    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:02:13.255649 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:13.255607    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:02:23.879722 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:02:23.879684    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:03:06.261375 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:03:06.261336    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:03:36.353070 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:03:36.353035    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:03:37.306247 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:03:37.306212    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:03:38.822059 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:03:38.822016    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:03:46.243673 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:03:46.243630    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:04:03.055398 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:04:03.055355    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:04:10.852101 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:04:10.852061    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:04:18.243742 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:04:18.243704    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:04:25.447145 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:04:25.447050    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:04:33.748196 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:04:33.748160    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:04:40.845671 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:04:40.845637    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:04:49.242200 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:04:49.242166    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:04:57.260911 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:04:57.260876    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:05:29.336758 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:05:29.336722    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 13:05:29.340808 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:05:29.340786    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rpkkg_371b2ab6-5733-4e72-80bf-36ddbdc2ea74/ovn-acl-logging/0.log"
Sep 15 13:05:47.201974 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:05:47.201933    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:05:47.251031 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:05:47.250999    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:06:24.348083 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:24.348048    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:06:29.542526 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:29.542487    2579 scope.go:117] "RemoveContainer" containerID="478373bb3e41fe1bb58f0d941c0b94c9481625d0b9a2046db3ce616f20d8a340"
Sep 15 13:06:31.347885 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:31.347848    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:06:41.356873 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:41.356823    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:06:42.450657 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.450622    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"]
Sep 15 13:06:42.451060 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.451038    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="136a2d5c-2cb7-4f35-8df6-d1eb52af7467" containerName="maas-api"
Sep 15 13:06:42.451060 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.451051    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="136a2d5c-2cb7-4f35-8df6-d1eb52af7467" containerName="maas-api"
Sep 15 13:06:42.451132 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.451065    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12dd8566-525c-4af9-84a6-418479e879aa" containerName="main"
Sep 15 13:06:42.451132 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.451071    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="12dd8566-525c-4af9-84a6-418479e879aa" containerName="main"
Sep 15 13:06:42.451132 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.451082    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a5ef640-3f07-4bab-9a20-bbf6423386ce" containerName="cleanup"
Sep 15 13:06:42.451132 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.451092    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a5ef640-3f07-4bab-9a20-bbf6423386ce" containerName="cleanup"
Sep 15 13:06:42.451132 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.451100    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b80481b6-a19b-447b-b6d2-2410d159369a" containerName="cleanup"
Sep 15 13:06:42.451132 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.451105    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b80481b6-a19b-447b-b6d2-2410d159369a" containerName="cleanup"
Sep 15 13:06:42.451132 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.451115    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79efd4dc-69e1-4262-b7d5-20914de57933" containerName="maas-api"
Sep 15 13:06:42.451132 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.451120    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79efd4dc-69e1-4262-b7d5-20914de57933" containerName="maas-api"
Sep 15 13:06:42.451376 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.451176    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79efd4dc-69e1-4262-b7d5-20914de57933" containerName="maas-api"
Sep 15 13:06:42.451376 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.451184    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="12dd8566-525c-4af9-84a6-418479e879aa" containerName="main"
Sep 15 13:06:42.451376 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.451190    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="136a2d5c-2cb7-4f35-8df6-d1eb52af7467" containerName="maas-api"
Sep 15 13:06:42.451376 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.451197    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a5ef640-3f07-4bab-9a20-bbf6423386ce" containerName="cleanup"
Sep 15 13:06:42.451376 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.451202    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b80481b6-a19b-447b-b6d2-2410d159369a" containerName="cleanup"
Sep 15 13:06:42.454282 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.454255    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.456731 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.456708    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b8c597a1-kserve-self-signed-certs\""
Sep 15 13:06:42.463795 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.463772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"]
Sep 15 13:06:42.486137 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.486106    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:06:42.538669 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.538633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f276031a-01ab-481a-8d9a-badda6cf4c57-tls-certs\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.538881 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.538676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-model-cache\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.538881 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.538704    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-dshm\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.538881 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.538783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-tmp-dir\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.538881 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.538822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-home\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.538881 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.538869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57q9s\" (UniqueName: \"kubernetes.io/projected/f276031a-01ab-481a-8d9a-badda6cf4c57-kube-api-access-57q9s\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.639950 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.639908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f276031a-01ab-481a-8d9a-badda6cf4c57-tls-certs\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.640123 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.639959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-model-cache\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.640123 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.639985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-dshm\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.640123 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.640006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-tmp-dir\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.640123 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.640029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-home\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.640123 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.640046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57q9s\" (UniqueName: \"kubernetes.io/projected/f276031a-01ab-481a-8d9a-badda6cf4c57-kube-api-access-57q9s\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.640517 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.640482    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-home\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.640517 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.640512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-model-cache\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.640665 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.640583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-tmp-dir\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.642305 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.642284    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-dshm\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.642560 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.642543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f276031a-01ab-481a-8d9a-badda6cf4c57-tls-certs\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.647664 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.647642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57q9s\" (UniqueName: \"kubernetes.io/projected/f276031a-01ab-481a-8d9a-badda6cf4c57-kube-api-access-57q9s\") pod \"e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") " pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:42.764909 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:42.764808    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:43.038941 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:43.038913    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"]
Sep 15 13:06:43.040601 ip-10-0-129-17 kubenswrapper[2579]: W0915 13:06:43.040572    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf276031a_01ab_481a_8d9a_badda6cf4c57.slice/crio-f3fb8f231cab69618cb0331e90af2200a0502a0fe90b1ca5818ac0a4f0a7aff0 WatchSource:0}: Error finding container f3fb8f231cab69618cb0331e90af2200a0502a0fe90b1ca5818ac0a4f0a7aff0: Status 404 returned error can't find the container with id f3fb8f231cab69618cb0331e90af2200a0502a0fe90b1ca5818ac0a4f0a7aff0
Sep 15 13:06:44.009724 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:44.009683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94" event={"ID":"f276031a-01ab-481a-8d9a-badda6cf4c57","Type":"ContainerStarted","Data":"8fcd3ff3908efa149be72146837075a3a177a54f8dcd5a1c8df48204566bc339"}
Sep 15 13:06:44.009724 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:44.009729    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94" event={"ID":"f276031a-01ab-481a-8d9a-badda6cf4c57","Type":"ContainerStarted","Data":"f3fb8f231cab69618cb0331e90af2200a0502a0fe90b1ca5818ac0a4f0a7aff0"}
Sep 15 13:06:44.027054 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:44.027006    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94" podStartSLOduration=2.026991533 podStartE2EDuration="2.026991533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:06:44.024547212 +0000 UTC m=+2475.409258110" watchObservedRunningTime="2026-09-15 13:06:44.026991533 +0000 UTC m=+2475.411702417"
Sep 15 13:06:47.075919 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:47.075884    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:06:49.253120 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:49.253078    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:06:51.386914 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:51.386878    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:06:52.765367 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:52.765318    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:52.765778 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:52.765385    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:52.778291 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:52.778265    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:53.068363 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:53.068277    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:06:55.263750 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:06:55.263697    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:07:34.638817 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:34.638741    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"]
Sep 15 13:07:34.659184 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:34.659157    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:07:36.054881 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:36.054825    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:07:37.850515 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:37.850479    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:07:46.447715 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:46.447680    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:07:49.658144 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:49.658092    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94" podUID="f276031a-01ab-481a-8d9a-badda6cf4c57" containerName="main" containerID="cri-o://8fcd3ff3908efa149be72146837075a3a177a54f8dcd5a1c8df48204566bc339" gracePeriod=45
Sep 15 13:07:49.911116 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:49.911059    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:07:50.057641 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.057602    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-home\") pod \"f276031a-01ab-481a-8d9a-badda6cf4c57\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") "
Sep 15 13:07:50.057826 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.057670    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f276031a-01ab-481a-8d9a-badda6cf4c57-tls-certs\") pod \"f276031a-01ab-481a-8d9a-badda6cf4c57\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") "
Sep 15 13:07:50.057826 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.057714    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-tmp-dir\") pod \"f276031a-01ab-481a-8d9a-badda6cf4c57\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") "
Sep 15 13:07:50.057826 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.057757    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-model-cache\") pod \"f276031a-01ab-481a-8d9a-badda6cf4c57\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") "
Sep 15 13:07:50.058028 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.057821    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57q9s\" (UniqueName: \"kubernetes.io/projected/f276031a-01ab-481a-8d9a-badda6cf4c57-kube-api-access-57q9s\") pod \"f276031a-01ab-481a-8d9a-badda6cf4c57\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") "
Sep 15 13:07:50.058028 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.057917    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-dshm\") pod \"f276031a-01ab-481a-8d9a-badda6cf4c57\" (UID: \"f276031a-01ab-481a-8d9a-badda6cf4c57\") "
Sep 15 13:07:50.058028 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.057920    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-home" (OuterVolumeSpecName: "home") pod "f276031a-01ab-481a-8d9a-badda6cf4c57" (UID: "f276031a-01ab-481a-8d9a-badda6cf4c57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:07:50.058213 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.058019    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f276031a-01ab-481a-8d9a-badda6cf4c57" (UID: "f276031a-01ab-481a-8d9a-badda6cf4c57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:07:50.058213 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.058052    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-model-cache" (OuterVolumeSpecName: "model-cache") pod "f276031a-01ab-481a-8d9a-badda6cf4c57" (UID: "f276031a-01ab-481a-8d9a-badda6cf4c57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:07:50.058213 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.058184    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-home\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:07:50.058213 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.058201    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-tmp-dir\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:07:50.058351 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.058213    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-model-cache\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:07:50.060070 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.060038    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-dshm" (OuterVolumeSpecName: "dshm") pod "f276031a-01ab-481a-8d9a-badda6cf4c57" (UID: "f276031a-01ab-481a-8d9a-badda6cf4c57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 13:07:50.060070 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.060040    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f276031a-01ab-481a-8d9a-badda6cf4c57-kube-api-access-57q9s" (OuterVolumeSpecName: "kube-api-access-57q9s") pod "f276031a-01ab-481a-8d9a-badda6cf4c57" (UID: "f276031a-01ab-481a-8d9a-badda6cf4c57"). InnerVolumeSpecName "kube-api-access-57q9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:07:50.060215 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.060089    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f276031a-01ab-481a-8d9a-badda6cf4c57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f276031a-01ab-481a-8d9a-badda6cf4c57" (UID: "f276031a-01ab-481a-8d9a-badda6cf4c57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:07:50.159575 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.159534    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f276031a-01ab-481a-8d9a-badda6cf4c57-tls-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:07:50.159575 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.159569    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57q9s\" (UniqueName: \"kubernetes.io/projected/f276031a-01ab-481a-8d9a-badda6cf4c57-kube-api-access-57q9s\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:07:50.159575 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.159580    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f276031a-01ab-481a-8d9a-badda6cf4c57-dshm\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:07:50.346526 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.346441    2579 generic.go:356] "Generic (PLEG): container finished" podID="f276031a-01ab-481a-8d9a-badda6cf4c57" containerID="8fcd3ff3908efa149be72146837075a3a177a54f8dcd5a1c8df48204566bc339" exitCode=0
Sep 15 13:07:50.346526 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.346484    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"
Sep 15 13:07:50.346526 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.346512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94" event={"ID":"f276031a-01ab-481a-8d9a-badda6cf4c57","Type":"ContainerDied","Data":"8fcd3ff3908efa149be72146837075a3a177a54f8dcd5a1c8df48204566bc339"}
Sep 15 13:07:50.346772 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.346544    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94" event={"ID":"f276031a-01ab-481a-8d9a-badda6cf4c57","Type":"ContainerDied","Data":"f3fb8f231cab69618cb0331e90af2200a0502a0fe90b1ca5818ac0a4f0a7aff0"}
Sep 15 13:07:50.346772 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.346560    2579 scope.go:117] "RemoveContainer" containerID="8fcd3ff3908efa149be72146837075a3a177a54f8dcd5a1c8df48204566bc339"
Sep 15 13:07:50.356365 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.356348    2579 scope.go:117] "RemoveContainer" containerID="8fcd3ff3908efa149be72146837075a3a177a54f8dcd5a1c8df48204566bc339"
Sep 15 13:07:50.356619 ip-10-0-129-17 kubenswrapper[2579]: E0915 13:07:50.356603    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fcd3ff3908efa149be72146837075a3a177a54f8dcd5a1c8df48204566bc339\": container with ID starting with 8fcd3ff3908efa149be72146837075a3a177a54f8dcd5a1c8df48204566bc339 not found: ID does not exist" containerID="8fcd3ff3908efa149be72146837075a3a177a54f8dcd5a1c8df48204566bc339"
Sep 15 13:07:50.356659 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.356627    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fcd3ff3908efa149be72146837075a3a177a54f8dcd5a1c8df48204566bc339"} err="failed to get container status \"8fcd3ff3908efa149be72146837075a3a177a54f8dcd5a1c8df48204566bc339\": rpc error: code = NotFound desc = could not find container \"8fcd3ff3908efa149be72146837075a3a177a54f8dcd5a1c8df48204566bc339\": container with ID starting with 8fcd3ff3908efa149be72146837075a3a177a54f8dcd5a1c8df48204566bc339 not found: ID does not exist"
Sep 15 13:07:50.367213 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.367187    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"]
Sep 15 13:07:50.371261 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:50.371238    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b8c597a1-kserve-78b796575f-z7p94"]
Sep 15 13:07:51.105484 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:51.105451    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f276031a-01ab-481a-8d9a-badda6cf4c57" path="/var/lib/kubelet/pods/f276031a-01ab-481a-8d9a-badda6cf4c57/volumes"
Sep 15 13:07:54.176869 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.176818    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"]
Sep 15 13:07:54.177278 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.177199    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f276031a-01ab-481a-8d9a-badda6cf4c57" containerName="main"
Sep 15 13:07:54.177278 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.177211    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f276031a-01ab-481a-8d9a-badda6cf4c57" containerName="main"
Sep 15 13:07:54.177278 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.177272    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f276031a-01ab-481a-8d9a-badda6cf4c57" containerName="main"
Sep 15 13:07:54.181697 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.181677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:07:54.185512 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.185484    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 13:07:54.185642 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.185538    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 13:07:54.185642 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.185489    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 13:07:54.185738 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.185655    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kqdbn\""
Sep 15 13:07:54.185938 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.185916    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 13:07:54.188319 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.188290    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"]
Sep 15 13:07:54.189268 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.188781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d450290-e2c2-4741-8279-ac9103f69b96-metrics-certs\") pod \"maas-api-556658ccdd-7pcfm\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") " pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:07:54.189268 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.188823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxwv7\" (UniqueName: \"kubernetes.io/projected/1d450290-e2c2-4741-8279-ac9103f69b96-kube-api-access-nxwv7\") pod \"maas-api-556658ccdd-7pcfm\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") " pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:07:54.189268 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.188869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d450290-e2c2-4741-8279-ac9103f69b96-maas-api-tls\") pod \"maas-api-556658ccdd-7pcfm\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") " pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:07:54.189268 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.188894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d450290-e2c2-4741-8279-ac9103f69b96-openshift-service-ca\") pod \"maas-api-556658ccdd-7pcfm\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") " pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:07:54.290057 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.290027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d450290-e2c2-4741-8279-ac9103f69b96-metrics-certs\") pod \"maas-api-556658ccdd-7pcfm\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") " pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:07:54.290259 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.290063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxwv7\" (UniqueName: \"kubernetes.io/projected/1d450290-e2c2-4741-8279-ac9103f69b96-kube-api-access-nxwv7\") pod \"maas-api-556658ccdd-7pcfm\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") " pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:07:54.290259 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.290082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d450290-e2c2-4741-8279-ac9103f69b96-maas-api-tls\") pod \"maas-api-556658ccdd-7pcfm\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") " pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:07:54.290259 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.290100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d450290-e2c2-4741-8279-ac9103f69b96-openshift-service-ca\") pod \"maas-api-556658ccdd-7pcfm\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") " pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:07:54.290772 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.290751    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d450290-e2c2-4741-8279-ac9103f69b96-openshift-service-ca\") pod \"maas-api-556658ccdd-7pcfm\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") " pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:07:54.292628 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.292596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d450290-e2c2-4741-8279-ac9103f69b96-metrics-certs\") pod \"maas-api-556658ccdd-7pcfm\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") " pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:07:54.292727 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.292637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d450290-e2c2-4741-8279-ac9103f69b96-maas-api-tls\") pod \"maas-api-556658ccdd-7pcfm\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") " pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:07:54.297768 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.297748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxwv7\" (UniqueName: \"kubernetes.io/projected/1d450290-e2c2-4741-8279-ac9103f69b96-kube-api-access-nxwv7\") pod \"maas-api-556658ccdd-7pcfm\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") " pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:07:54.493462 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.493366    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:07:54.750745 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.750670    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:07:54.770104 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.770069    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"]
Sep 15 13:07:54.772307 ip-10-0-129-17 kubenswrapper[2579]: W0915 13:07:54.772279    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d450290_e2c2_4741_8279_ac9103f69b96.slice/crio-1b9265bdbb48b4b4c2f1364233cf7b7725d2404da4c24fb346cc304c7694aeda WatchSource:0}: Error finding container 1b9265bdbb48b4b4c2f1364233cf7b7725d2404da4c24fb346cc304c7694aeda: Status 404 returned error can't find the container with id 1b9265bdbb48b4b4c2f1364233cf7b7725d2404da4c24fb346cc304c7694aeda
Sep 15 13:07:54.773636 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:54.773618    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 13:07:55.374474 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:55.374443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm" event={"ID":"1d450290-e2c2-4741-8279-ac9103f69b96","Type":"ContainerStarted","Data":"fc92b7636cb32c6c6d6f8351aee8c8ecb9335199edd425a183b4cd265f03ee91"}
Sep 15 13:07:55.374906 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:55.374495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm" event={"ID":"1d450290-e2c2-4741-8279-ac9103f69b96","Type":"ContainerStarted","Data":"1b9265bdbb48b4b4c2f1364233cf7b7725d2404da4c24fb346cc304c7694aeda"}
Sep 15 13:07:55.374906 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:55.374567    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:07:55.390158 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:07:55.390107    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm" podStartSLOduration=0.892821525 podStartE2EDuration="1.390090688s" totalImagesPullingTime="497.269163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:07:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:07:55.389734118 +0000 UTC m=+2546.774445001" watchObservedRunningTime="2026-09-15 13:07:55.390090688 +0000 UTC m=+2546.774801599"
Sep 15 13:08:01.384951 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:01.384922    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:08:01.851274 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:01.851242    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:08:02.416932 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.416898    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-576799898d-xfvts"]
Sep 15 13:08:02.420380 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.420363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:02.430157 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.430127    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-576799898d-xfvts"]
Sep 15 13:08:02.463528 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.463488    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e2fdb85-7cc2-4467-b723-4c23340a0ee0-maas-api-tls\") pod \"maas-api-576799898d-xfvts\" (UID: \"9e2fdb85-7cc2-4467-b723-4c23340a0ee0\") " pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:02.463714 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.463531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g59ml\" (UniqueName: \"kubernetes.io/projected/9e2fdb85-7cc2-4467-b723-4c23340a0ee0-kube-api-access-g59ml\") pod \"maas-api-576799898d-xfvts\" (UID: \"9e2fdb85-7cc2-4467-b723-4c23340a0ee0\") " pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:02.463714 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.463642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e2fdb85-7cc2-4467-b723-4c23340a0ee0-openshift-service-ca\") pod \"maas-api-576799898d-xfvts\" (UID: \"9e2fdb85-7cc2-4467-b723-4c23340a0ee0\") " pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:02.463828 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.463742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e2fdb85-7cc2-4467-b723-4c23340a0ee0-metrics-certs\") pod \"maas-api-576799898d-xfvts\" (UID: \"9e2fdb85-7cc2-4467-b723-4c23340a0ee0\") " pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:02.564757 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.564717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e2fdb85-7cc2-4467-b723-4c23340a0ee0-openshift-service-ca\") pod \"maas-api-576799898d-xfvts\" (UID: \"9e2fdb85-7cc2-4467-b723-4c23340a0ee0\") " pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:02.564980 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.564784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e2fdb85-7cc2-4467-b723-4c23340a0ee0-metrics-certs\") pod \"maas-api-576799898d-xfvts\" (UID: \"9e2fdb85-7cc2-4467-b723-4c23340a0ee0\") " pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:02.564980 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.564815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e2fdb85-7cc2-4467-b723-4c23340a0ee0-maas-api-tls\") pod \"maas-api-576799898d-xfvts\" (UID: \"9e2fdb85-7cc2-4467-b723-4c23340a0ee0\") " pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:02.564980 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.564859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g59ml\" (UniqueName: \"kubernetes.io/projected/9e2fdb85-7cc2-4467-b723-4c23340a0ee0-kube-api-access-g59ml\") pod \"maas-api-576799898d-xfvts\" (UID: \"9e2fdb85-7cc2-4467-b723-4c23340a0ee0\") " pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:02.565499 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.565474    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e2fdb85-7cc2-4467-b723-4c23340a0ee0-openshift-service-ca\") pod \"maas-api-576799898d-xfvts\" (UID: \"9e2fdb85-7cc2-4467-b723-4c23340a0ee0\") " pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:02.567345 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.567325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e2fdb85-7cc2-4467-b723-4c23340a0ee0-metrics-certs\") pod \"maas-api-576799898d-xfvts\" (UID: \"9e2fdb85-7cc2-4467-b723-4c23340a0ee0\") " pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:02.567420 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.567386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e2fdb85-7cc2-4467-b723-4c23340a0ee0-maas-api-tls\") pod \"maas-api-576799898d-xfvts\" (UID: \"9e2fdb85-7cc2-4467-b723-4c23340a0ee0\") " pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:02.572303 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.572281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g59ml\" (UniqueName: \"kubernetes.io/projected/9e2fdb85-7cc2-4467-b723-4c23340a0ee0-kube-api-access-g59ml\") pod \"maas-api-576799898d-xfvts\" (UID: \"9e2fdb85-7cc2-4467-b723-4c23340a0ee0\") " pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:02.732036 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:02.731930    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:03.007228 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:03.007197    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-576799898d-xfvts"]
Sep 15 13:08:03.008454 ip-10-0-129-17 kubenswrapper[2579]: W0915 13:08:03.008428    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2fdb85_7cc2_4467_b723_4c23340a0ee0.slice/crio-d474acdd5742620027e8f8e1f03544b828662cf0456b6404dfbad87375d0b2c3 WatchSource:0}: Error finding container d474acdd5742620027e8f8e1f03544b828662cf0456b6404dfbad87375d0b2c3: Status 404 returned error can't find the container with id d474acdd5742620027e8f8e1f03544b828662cf0456b6404dfbad87375d0b2c3
Sep 15 13:08:03.416231 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:03.416196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts" event={"ID":"9e2fdb85-7cc2-4467-b723-4c23340a0ee0","Type":"ContainerStarted","Data":"d474acdd5742620027e8f8e1f03544b828662cf0456b6404dfbad87375d0b2c3"}
Sep 15 13:08:06.432297 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:06.432255    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts" event={"ID":"9e2fdb85-7cc2-4467-b723-4c23340a0ee0","Type":"ContainerStarted","Data":"c01ff62f256d5515505e8176b83c0d21fc91f04bdebd1ae52369e41fc615cd7b"}
Sep 15 13:08:06.432719 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:06.432323    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:06.449775 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:06.449721    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts" podStartSLOduration=1.150831224 podStartE2EDuration="4.449703572s" totalImagesPullingTime="3.298872348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:08:06.447628689 +0000 UTC m=+2557.832339573" watchObservedRunningTime="2026-09-15 13:08:06.449703572 +0000 UTC m=+2557.834414496"
Sep 15 13:08:08.949611 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:08.949577    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:08:12.443364 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:12.443332    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-576799898d-xfvts"
Sep 15 13:08:12.489883 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:12.489850    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"]
Sep 15 13:08:12.490120 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:12.490096    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm" podUID="1d450290-e2c2-4741-8279-ac9103f69b96" containerName="maas-api" containerID="cri-o://fc92b7636cb32c6c6d6f8351aee8c8ecb9335199edd425a183b4cd265f03ee91" gracePeriod=30
Sep 15 13:08:13.737916 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:13.737894    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:08:13.871780 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:13.871684    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d450290-e2c2-4741-8279-ac9103f69b96-maas-api-tls\") pod \"1d450290-e2c2-4741-8279-ac9103f69b96\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") "
Sep 15 13:08:13.871780 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:13.871738    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d450290-e2c2-4741-8279-ac9103f69b96-openshift-service-ca\") pod \"1d450290-e2c2-4741-8279-ac9103f69b96\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") "
Sep 15 13:08:13.871780 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:13.871785    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d450290-e2c2-4741-8279-ac9103f69b96-metrics-certs\") pod \"1d450290-e2c2-4741-8279-ac9103f69b96\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") "
Sep 15 13:08:13.872070 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:13.871880    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxwv7\" (UniqueName: \"kubernetes.io/projected/1d450290-e2c2-4741-8279-ac9103f69b96-kube-api-access-nxwv7\") pod \"1d450290-e2c2-4741-8279-ac9103f69b96\" (UID: \"1d450290-e2c2-4741-8279-ac9103f69b96\") "
Sep 15 13:08:13.872151 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:13.872121    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d450290-e2c2-4741-8279-ac9103f69b96-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1d450290-e2c2-4741-8279-ac9103f69b96" (UID: "1d450290-e2c2-4741-8279-ac9103f69b96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 13:08:13.874090 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:13.874065    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d450290-e2c2-4741-8279-ac9103f69b96-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1d450290-e2c2-4741-8279-ac9103f69b96" (UID: "1d450290-e2c2-4741-8279-ac9103f69b96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:08:13.874090 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:13.874079    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d450290-e2c2-4741-8279-ac9103f69b96-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1d450290-e2c2-4741-8279-ac9103f69b96" (UID: "1d450290-e2c2-4741-8279-ac9103f69b96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 13:08:13.874240 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:13.874161    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d450290-e2c2-4741-8279-ac9103f69b96-kube-api-access-nxwv7" (OuterVolumeSpecName: "kube-api-access-nxwv7") pod "1d450290-e2c2-4741-8279-ac9103f69b96" (UID: "1d450290-e2c2-4741-8279-ac9103f69b96"). InnerVolumeSpecName "kube-api-access-nxwv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 13:08:13.973020 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:13.972979    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxwv7\" (UniqueName: \"kubernetes.io/projected/1d450290-e2c2-4741-8279-ac9103f69b96-kube-api-access-nxwv7\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:08:13.973020 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:13.973013    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d450290-e2c2-4741-8279-ac9103f69b96-maas-api-tls\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:08:13.973020 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:13.973027    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d450290-e2c2-4741-8279-ac9103f69b96-openshift-service-ca\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:08:13.973293 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:13.973040    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d450290-e2c2-4741-8279-ac9103f69b96-metrics-certs\") on node \"ip-10-0-129-17.ec2.internal\" DevicePath \"\""
Sep 15 13:08:14.474892 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:14.474860    2579 generic.go:356] "Generic (PLEG): container finished" podID="1d450290-e2c2-4741-8279-ac9103f69b96" containerID="fc92b7636cb32c6c6d6f8351aee8c8ecb9335199edd425a183b4cd265f03ee91" exitCode=0
Sep 15 13:08:14.474892 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:14.474892    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"
Sep 15 13:08:14.475129 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:14.474926    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm" event={"ID":"1d450290-e2c2-4741-8279-ac9103f69b96","Type":"ContainerDied","Data":"fc92b7636cb32c6c6d6f8351aee8c8ecb9335199edd425a183b4cd265f03ee91"}
Sep 15 13:08:14.475129 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:14.474978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm" event={"ID":"1d450290-e2c2-4741-8279-ac9103f69b96","Type":"ContainerDied","Data":"1b9265bdbb48b4b4c2f1364233cf7b7725d2404da4c24fb346cc304c7694aeda"}
Sep 15 13:08:14.475129 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:14.474999    2579 scope.go:117] "RemoveContainer" containerID="fc92b7636cb32c6c6d6f8351aee8c8ecb9335199edd425a183b4cd265f03ee91"
Sep 15 13:08:14.484908 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:14.484891    2579 scope.go:117] "RemoveContainer" containerID="fc92b7636cb32c6c6d6f8351aee8c8ecb9335199edd425a183b4cd265f03ee91"
Sep 15 13:08:14.485202 ip-10-0-129-17 kubenswrapper[2579]: E0915 13:08:14.485177    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc92b7636cb32c6c6d6f8351aee8c8ecb9335199edd425a183b4cd265f03ee91\": container with ID starting with fc92b7636cb32c6c6d6f8351aee8c8ecb9335199edd425a183b4cd265f03ee91 not found: ID does not exist" containerID="fc92b7636cb32c6c6d6f8351aee8c8ecb9335199edd425a183b4cd265f03ee91"
Sep 15 13:08:14.485253 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:14.485212    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc92b7636cb32c6c6d6f8351aee8c8ecb9335199edd425a183b4cd265f03ee91"} err="failed to get container status \"fc92b7636cb32c6c6d6f8351aee8c8ecb9335199edd425a183b4cd265f03ee91\": rpc error: code = NotFound desc = could not find container \"fc92b7636cb32c6c6d6f8351aee8c8ecb9335199edd425a183b4cd265f03ee91\": container with ID starting with fc92b7636cb32c6c6d6f8351aee8c8ecb9335199edd425a183b4cd265f03ee91 not found: ID does not exist"
Sep 15 13:08:14.497045 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:14.497020    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"]
Sep 15 13:08:14.500374 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:14.500350    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-556658ccdd-7pcfm"]
Sep 15 13:08:15.105040 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:15.105008    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d450290-e2c2-4741-8279-ac9103f69b96" path="/var/lib/kubelet/pods/1d450290-e2c2-4741-8279-ac9103f69b96/volumes"
Sep 15 13:08:17.557043 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:17.557003    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:08:20.598176 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:20.598147    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:08:21.853275 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:21.853230    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tfv49"]
Sep 15 13:08:25.174188 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:25.174157    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66cfddbbb4-hnrps_c09f66d1-d1b7-40f3-bde9-3d226721da78/authorino/0.log"
Sep 15 13:08:31.335189 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:31.335158    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-665cd58c4f-hb9xb_bdb03845-14e9-4cf4-a957-6fc95cade506/copy-manifests/0.log"
Sep 15 13:08:31.347760 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:31.347730    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-665cd58c4f-hb9xb_bdb03845-14e9-4cf4-a957-6fc95cade506/manager/0.log"
Sep 15 13:08:31.933112 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:31.933080    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-576799898d-xfvts_9e2fdb85-7cc2-4467-b723-4c23340a0ee0/maas-api/0.log"
Sep 15 13:08:32.040626 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:32.040596    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29824605-9j559_ed29b541-4aac-4143-b8fd-6c0b6c160feb/cleanup/0.log"
Sep 15 13:08:32.154008 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:32.153982    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29824620-8t6hx_b80481b6-a19b-447b-b6d2-2410d159369a/cleanup/0.log"
Sep 15 13:08:33.027862 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:33.027812    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66cfddbbb4-hnrps_c09f66d1-d1b7-40f3-bde9-3d226721da78/authorino/0.log"
Sep 15 13:08:33.358270 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:33.358246    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-nrbwv_91f8fa70-291c-4441-a53b-4260034f4fa8/kuadrant-console-plugin/0.log"
Sep 15 13:08:33.629237 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:33.629155    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-tfv49_431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196/limitador/0.log"
Sep 15 13:08:34.331642 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:34.331608    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6956d8dd68-5w4fx_42f652d8-12b5-4295-8281-5e27afe32abc/kube-auth-proxy/0.log"
Sep 15 13:08:35.268942 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:35.268907    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-xw7fg_9b90415b-85bf-4314-96be-eae0f3af4e03/main/0.log"
Sep 15 13:08:35.733889 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:35.733862    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fd8w6v_69ff1e20-76e5-4178-b1e1-dd9858f21baf/main/0.log"
Sep 15 13:08:35.961434 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:35.961407    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-8pgwc_e9cb419d-9724-42b0-ac60-e05b675ce950/main/0.log"
Sep 15 13:08:41.995624 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:41.995584    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:08:42.064950 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:42.064909    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log" limit=1024
Sep 15 13:08:44.891469 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:44.891439    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6txlk_f2f04d5a-e442-4cf8-ba3a-1b37b7934fb1/global-pull-secret-syncer/0.log"
Sep 15 13:08:44.973616 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:44.973582    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4pc2d_e17c9098-956f-4d77-a535-5b09749f3a95/konnectivity-agent/0.log"
Sep 15 13:08:45.062938 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:45.062913    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-17.ec2.internal_0eab95c6c565e847818c125e64a2e65f/haproxy/0.log"
Sep 15 13:08:50.218349 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:50.218310    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66cfddbbb4-hnrps_c09f66d1-d1b7-40f3-bde9-3d226721da78/authorino/0.log"
Sep 15 13:08:50.298207 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:50.298178    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-nrbwv_91f8fa70-291c-4441-a53b-4260034f4fa8/kuadrant-console-plugin/0.log"
Sep 15 13:08:50.368273 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:50.368239    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-tfv49_431cdb7b-b8eb-4c2f-bdbd-4a0cb657b196/limitador/0.log"
Sep 15 13:08:52.314235 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:52.314085    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-88w75_0aeb0879-d842-44e4-a4ec-4494c5d7e48d/node-exporter/0.log"
Sep 15 13:08:52.339456 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:52.339433    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-88w75_0aeb0879-d842-44e4-a4ec-4494c5d7e48d/kube-rbac-proxy/0.log"
Sep 15 13:08:52.364030 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:52.364012    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-88w75_0aeb0879-d842-44e4-a4ec-4494c5d7e48d/init-textfile/0.log"
Sep 15 13:08:52.574925 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:52.574843    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-j7njs_9de9d475-0445-47d4-9b02-5f92e1bc1a85/kube-rbac-proxy-main/0.log"
Sep 15 13:08:52.596943 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:52.596915    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-j7njs_9de9d475-0445-47d4-9b02-5f92e1bc1a85/kube-rbac-proxy-self/0.log"
Sep 15 13:08:52.619538 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:52.619513    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-j7njs_9de9d475-0445-47d4-9b02-5f92e1bc1a85/openshift-state-metrics/0.log"
Sep 15 13:08:53.005653 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:53.005624    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68d4868d9c-prhnf_51582373-a74d-4cc0-893e-78ad4df5ec9e/thanos-query/0.log"
Sep 15 13:08:53.029981 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:53.029948    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68d4868d9c-prhnf_51582373-a74d-4cc0-893e-78ad4df5ec9e/kube-rbac-proxy-web/0.log"
Sep 15 13:08:53.052168 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:53.052145    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68d4868d9c-prhnf_51582373-a74d-4cc0-893e-78ad4df5ec9e/kube-rbac-proxy/0.log"
Sep 15 13:08:53.076402 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:53.076377    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68d4868d9c-prhnf_51582373-a74d-4cc0-893e-78ad4df5ec9e/prom-label-proxy/0.log"
Sep 15 13:08:53.097706 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:53.097684    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68d4868d9c-prhnf_51582373-a74d-4cc0-893e-78ad4df5ec9e/kube-rbac-proxy-rules/0.log"
Sep 15 13:08:53.118735 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:53.118712    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68d4868d9c-prhnf_51582373-a74d-4cc0-893e-78ad4df5ec9e/kube-rbac-proxy-metrics/0.log"
Sep 15 13:08:55.444960 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.444921    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"]
Sep 15 13:08:55.445488 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.445396    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d450290-e2c2-4741-8279-ac9103f69b96" containerName="maas-api"
Sep 15 13:08:55.445488 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.445411    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d450290-e2c2-4741-8279-ac9103f69b96" containerName="maas-api"
Sep 15 13:08:55.445623 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.445495    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d450290-e2c2-4741-8279-ac9103f69b96" containerName="maas-api"
Sep 15 13:08:55.448756 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.448738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.451284 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.451262    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fpbvp\"/\"default-dockercfg-kswxq\""
Sep 15 13:08:55.451405 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.451265    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fpbvp\"/\"openshift-service-ca.crt\""
Sep 15 13:08:55.452681 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.452664    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fpbvp\"/\"kube-root-ca.crt\""
Sep 15 13:08:55.459137 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.459113    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"]
Sep 15 13:08:55.541491 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.541458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2tb\" (UniqueName: \"kubernetes.io/projected/17b0a8b7-ac97-4b45-844b-cbff6aa73b71-kube-api-access-xd2tb\") pod \"perf-node-gather-daemonset-mfhml\" (UID: \"17b0a8b7-ac97-4b45-844b-cbff6aa73b71\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.541704 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.541513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17b0a8b7-ac97-4b45-844b-cbff6aa73b71-podres\") pod \"perf-node-gather-daemonset-mfhml\" (UID: \"17b0a8b7-ac97-4b45-844b-cbff6aa73b71\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.541704 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.541550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17b0a8b7-ac97-4b45-844b-cbff6aa73b71-lib-modules\") pod \"perf-node-gather-daemonset-mfhml\" (UID: \"17b0a8b7-ac97-4b45-844b-cbff6aa73b71\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.541704 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.541578    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17b0a8b7-ac97-4b45-844b-cbff6aa73b71-sys\") pod \"perf-node-gather-daemonset-mfhml\" (UID: \"17b0a8b7-ac97-4b45-844b-cbff6aa73b71\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.541704 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.541595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17b0a8b7-ac97-4b45-844b-cbff6aa73b71-proc\") pod \"perf-node-gather-daemonset-mfhml\" (UID: \"17b0a8b7-ac97-4b45-844b-cbff6aa73b71\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.642995 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.642961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17b0a8b7-ac97-4b45-844b-cbff6aa73b71-sys\") pod \"perf-node-gather-daemonset-mfhml\" (UID: \"17b0a8b7-ac97-4b45-844b-cbff6aa73b71\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.642995 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.642996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17b0a8b7-ac97-4b45-844b-cbff6aa73b71-proc\") pod \"perf-node-gather-daemonset-mfhml\" (UID: \"17b0a8b7-ac97-4b45-844b-cbff6aa73b71\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.643201 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.643044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2tb\" (UniqueName: \"kubernetes.io/projected/17b0a8b7-ac97-4b45-844b-cbff6aa73b71-kube-api-access-xd2tb\") pod \"perf-node-gather-daemonset-mfhml\" (UID: \"17b0a8b7-ac97-4b45-844b-cbff6aa73b71\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.643201 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.643081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17b0a8b7-ac97-4b45-844b-cbff6aa73b71-podres\") pod \"perf-node-gather-daemonset-mfhml\" (UID: \"17b0a8b7-ac97-4b45-844b-cbff6aa73b71\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.643201 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.643099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17b0a8b7-ac97-4b45-844b-cbff6aa73b71-lib-modules\") pod \"perf-node-gather-daemonset-mfhml\" (UID: \"17b0a8b7-ac97-4b45-844b-cbff6aa73b71\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.643201 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.643098    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17b0a8b7-ac97-4b45-844b-cbff6aa73b71-sys\") pod \"perf-node-gather-daemonset-mfhml\" (UID: \"17b0a8b7-ac97-4b45-844b-cbff6aa73b71\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.643201 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.643138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17b0a8b7-ac97-4b45-844b-cbff6aa73b71-proc\") pod \"perf-node-gather-daemonset-mfhml\" (UID: \"17b0a8b7-ac97-4b45-844b-cbff6aa73b71\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.643370 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.643217    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17b0a8b7-ac97-4b45-844b-cbff6aa73b71-lib-modules\") pod \"perf-node-gather-daemonset-mfhml\" (UID: \"17b0a8b7-ac97-4b45-844b-cbff6aa73b71\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.643370 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.643236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17b0a8b7-ac97-4b45-844b-cbff6aa73b71-podres\") pod \"perf-node-gather-daemonset-mfhml\" (UID: \"17b0a8b7-ac97-4b45-844b-cbff6aa73b71\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.650791 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.650759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2tb\" (UniqueName: \"kubernetes.io/projected/17b0a8b7-ac97-4b45-844b-cbff6aa73b71-kube-api-access-xd2tb\") pod \"perf-node-gather-daemonset-mfhml\" (UID: \"17b0a8b7-ac97-4b45-844b-cbff6aa73b71\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:55.762581 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:55.762488    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:56.032251 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:56.032224    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"]
Sep 15 13:08:56.033679 ip-10-0-129-17 kubenswrapper[2579]: W0915 13:08:56.033645    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod17b0a8b7_ac97_4b45_844b_cbff6aa73b71.slice/crio-d4479bf2112e813ab084c4166570cc8b1dadc433e9845e406d02cab72e104387 WatchSource:0}: Error finding container d4479bf2112e813ab084c4166570cc8b1dadc433e9845e406d02cab72e104387: Status 404 returned error can't find the container with id d4479bf2112e813ab084c4166570cc8b1dadc433e9845e406d02cab72e104387
Sep 15 13:08:56.694128 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:56.694091    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml" event={"ID":"17b0a8b7-ac97-4b45-844b-cbff6aa73b71","Type":"ContainerStarted","Data":"850da021178f6dc78585e529a79407e634063115c8957f97cba1426164b5d84e"}
Sep 15 13:08:56.694557 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:56.694157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml" event={"ID":"17b0a8b7-ac97-4b45-844b-cbff6aa73b71","Type":"ContainerStarted","Data":"d4479bf2112e813ab084c4166570cc8b1dadc433e9845e406d02cab72e104387"}
Sep 15 13:08:56.694557 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:56.694221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:08:56.710363 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:56.710321    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml" podStartSLOduration=1.710306729 podStartE2EDuration="1.710306729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 13:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:08:56.708064635 +0000 UTC m=+2608.092775519" watchObservedRunningTime="2026-09-15 13:08:56.710306729 +0000 UTC m=+2608.095017629"
Sep 15 13:08:57.410431 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:57.410403    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2ndsn_7812e2fd-cb28-4988-a3e0-5d9433a853f1/dns/0.log"
Sep 15 13:08:57.431625 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:57.431599    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2ndsn_7812e2fd-cb28-4988-a3e0-5d9433a853f1/kube-rbac-proxy/0.log"
Sep 15 13:08:57.544118 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:57.544084    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-66lhz_866085d7-f0ed-4104-bd66-d5964b6619c9/dns-node-resolver/0.log"
Sep 15 13:08:58.120771 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:58.120741    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79b48c6d55-27vqd_9a90a1fc-7dd9-4350-9ead-8fce38cb37bf/registry/0.log"
Sep 15 13:08:58.141045 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:58.141021    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-blcfn_1c61e9c1-fd93-4f6b-ac81-1ac9273d1ecc/node-ca/0.log"
Sep 15 13:08:59.984176 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:08:59.984149    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6956d8dd68-5w4fx_42f652d8-12b5-4295-8281-5e27afe32abc/kube-auth-proxy/0.log"
Sep 15 13:09:01.076232 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:01.076206    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bx6hs_40dd46b2-753c-40ea-a811-15944e4951b0/serve-healthcheck-canary/0.log"
Sep 15 13:09:01.574494 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:01.574455    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-qccwt_49823d24-5865-4a85-bacb-195339de36b2/insights-operator/0.log"
Sep 15 13:09:01.670530 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:01.670506    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5stdh_a047387b-02b3-4291-bc1c-e61fedc7aea9/kube-rbac-proxy/0.log"
Sep 15 13:09:01.691809 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:01.691779    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5stdh_a047387b-02b3-4291-bc1c-e61fedc7aea9/exporter/0.log"
Sep 15 13:09:01.712691 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:01.712665    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5stdh_a047387b-02b3-4291-bc1c-e61fedc7aea9/extractor/0.log"
Sep 15 13:09:02.708266 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:02.708238    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-mfhml"
Sep 15 13:09:04.613579 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:04.613540    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-665cd58c4f-hb9xb_bdb03845-14e9-4cf4-a957-6fc95cade506/manager/0.log"
Sep 15 13:09:04.630462 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:04.630438    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-665cd58c4f-hb9xb_bdb03845-14e9-4cf4-a957-6fc95cade506/copy-manifests/0.log"
Sep 15 13:09:06.027178 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:06.027150    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-b47d9b998-9vlr6_34d45520-8fca-4d9b-9e36-b6b16b727e8c/manager/0.log"
Sep 15 13:09:06.220086 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:06.220042    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-2kf95_5066a2c0-78da-4d6e-ad65-b97806371ff6/openshift-lws-operator/0.log"
Sep 15 13:09:06.823023 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:06.822934    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6956d8dd68-5w4fx_42f652d8-12b5-4295-8281-5e27afe32abc/kube-auth-proxy/0.log"
Sep 15 13:09:09.251849 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:09.251797    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6956d8dd68-5w4fx_42f652d8-12b5-4295-8281-5e27afe32abc/kube-auth-proxy/0.log"
Sep 15 13:09:11.387961 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:11.387900    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-fmtjg_4f01962c-d997-4add-9e2b-0a46caea55d0/migrator/0.log"
Sep 15 13:09:11.408096 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:11.408065    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-fmtjg_4f01962c-d997-4add-9e2b-0a46caea55d0/graceful-termination/0.log"
Sep 15 13:09:12.705348 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:12.705318    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ht8cc_6c05cc62-0238-4991-96af-7633ec6f0e05/kube-multus-additional-cni-plugins/0.log"
Sep 15 13:09:12.733156 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:12.733131    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ht8cc_6c05cc62-0238-4991-96af-7633ec6f0e05/egress-router-binary-copy/0.log"
Sep 15 13:09:12.769093 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:12.769062    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ht8cc_6c05cc62-0238-4991-96af-7633ec6f0e05/cni-plugins/0.log"
Sep 15 13:09:12.805131 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:12.805103    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ht8cc_6c05cc62-0238-4991-96af-7633ec6f0e05/bond-cni-plugin/0.log"
Sep 15 13:09:12.840093 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:12.840054    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ht8cc_6c05cc62-0238-4991-96af-7633ec6f0e05/routeoverride-cni/0.log"
Sep 15 13:09:12.880231 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:12.880184    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ht8cc_6c05cc62-0238-4991-96af-7633ec6f0e05/whereabouts-cni-bincopy/0.log"
Sep 15 13:09:12.914268 ip-10-0-129-17 kubenswrapper[2579]: I0915 13:09:12.914232    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ht8cc_6c05cc62-0238-4991-96af-7633ec6f0e05/whereabouts-cni/0.log"